AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |
Back to Blog
Redshift copy4/27/2023 At this point we adopted an easy upgrade: instead of hosting our own MySQL server, we loaded the data into Amazon Aurora. Eventually we saw more and more timed-out queries. All those factors took a toll on our little MySQL server. In the meantime our OLTP data becomes exponentially larger, which we certainly won’t complain. As PeriscopeData enabled more people to fiddle with the data, our data visualizations and queries increased in both numbers and complexity. This simple setup worked perfectly at the beginning. We run the two sets of scripts periodically using Jenkins. The second set of scripts batch insert data from CSV into MySQL database. Hence, in this iteration, we have created two sets of scripts: one set exports MongoDB’s data into CSV files by collection, or CSV files that describe many-to-many relationships. However, we’d like to do a few data transformations and cleansing before loading it into the analytical database. MongoDB has a mongoexport utility that can export MongoDB collections into CSV or JSON files. In our first ETL iteration, we setup a MySQL server as our analytics database. PeriscopeData works with most SQL databases. All you have to do is writing a few lines of SQL statements and a couple clicks. It makes creating beautiful data visualizations and auto-updating dashboards dead easy for anyone. To make our data more accessible for not only developers, but also product managers, designers, sales, and marketing people, we have chosen to integrate with an analytic tool PeriscopeData. What we needed is an analytical database. However, its API is not ideal to run complex or ad-hoc analytical queries. We chose MongoDB as our OLTP database for its flexibility and amazing JavaScript API. In this article I’ll describe what our current data pipeline setup is to support this goal. At Battlefy we try our best to make our data as transparent as possible across the whole organization. ![]() ![]() ![]() You Can’t Improve What you Don’t Measure.
0 Comments
Read More
Leave a Reply. |