Ingestion from Bloomberg

Task

We have daily_index_data_bloomberg_exporter periodic task that updates "Index States" table where we store tickers' daily values.
daily_index_data_bloomberg_exporter periodic task executes according code that runs Bloomberg exporter API client, process response file (it takes about 20 minutes) and puts obtained data into our database. As the documentation says the number of data points is limited by 30000 per request file.

Schedule

The task is scheduled to run every day of week from Monday through Friday at 1 p.m. (London time).
If we need data in our database immediately then we can run this periodic task manually.

Tickers

There is the "Index" table in our database populated with ticker names according to Bloomberg API. So if we need to get from Bloomberg data for the ticker that is not in our database the only thing that has to be done is to add the index of interest in this "Index" table using Django admin. Thus we expand the list of tickers in our database.

IP Whitelisting

Bloomberg requires IP whitelisting so we rent a static IP address from Heroku's Proximo plugin. It costs $25/month.

Monitoring

We monitor the Bloomberg celery worker in the Flower.


Code

GitHub