For Pixel City, an in development social experience, I create and manage the external backend called Talaris. Talaris is responsible for:

• Scalably receiving requests from many Roblox game instances
• Authorizing requests only from own servers
• Providing OpenAPI documentation
• Interacting with the Roblox web API through many proxies
• Caching important data to ensure low latency
• Serving user bought ads in-game based on a bidding system
• Compiling analytics of outfit usage and search statistics
• Search functionality for user outfits, by tags, name, description, and filters

Roblox-to-backend client

To integrate Talaris into the backend of the Roblox game instances, a client for the API was made, that automatically generates TS types from the OpenAPI definition. It is responsible for communicating with Talaris using HTTPService and authenticating requests.

Ratelimit system

During the development of this project, the HTTPService 500 requests per minute per server restriction was one of the most important considerations for the design of this system. Using a naive design, this limit would have been surpassed easily because of the high player count servers.

A Roblox side ratelimiting solution was developed, using a sliding log algorithm, with support for multiple buckets, and the ability to consume multiple ratelimit categories at once while returning accurate Retry-After timestamps. This solution is deployed at a RemoteEvent level.

Furthermore, time insensitive requests are batched into a per-minute ‘heartbeat’ loop to communicate with Talaris

Logging and dashboard

To monitor various system statistics, such as request latency, most used API paths, ad count, and other player initiated behavior, a Grafana dashboard is employed, using Prometheus to interact with the data, both self hosted.

For controlling the ad economy and doing other manual adjustments, an AppSmith instance is hosted.

For aggregating logs, Grafana Loki is used. This way, we can track the logs of every API call in case of issues.

Ad server

Talaris features a purpose-built ad server wherein users are able to buy ads on UGC content for an inputted amount of credits. Credits are used to buy ad views, on an exchange rate that varies by demand.

Afterwards, ad campaigns are distributed to each connected game instance equally, with a system in place to prevent overserving. Redis (KeyDB implementation) is used to track the progress of ad campaigns.

Each ad campaign has detailed analytics information associated to it. For every ad serve, we are able to track how long the ad has been shown on screen, whether it has been clicked, along with player information, ultimately funelling into the purchase rate.

These statistics are displayed in-game, on advertisers’ management interface, allowing them to analyze how well the camapaign performed.