In response to the current outages Body.io has skilled, I wished to personally acknowledge the problems you’ve encountered and transparently share the steps we’re engaged on to enhance the scenario.
First, it goes with out saying that our product’s uptime and availability is an absolute high precedence. We all know lots of our prospects rely on Body.io for time-sensitive, mission-critical work.
Over the course of the previous a number of months, the variety of downtime incidents we’ve been experiencing has elevated and has resulted in a number of outages which have generally lasted for hours. That is utterly unacceptable.
What’s been occurring?
The size that our product must deal with has steadily elevated over time and sharply elevated over the previous 12 months. A few of the early architectural patterns of our product weren’t capable of deal with this new scale and we’ve been onerous at work on rebuilding vital elements of our programs.
Behind the scenes, the Body.io engineering staff has been engaged on architecting important enhancements to our backend, which can put together the product for the following degree of scale and stop the downtime incidents we’ve been experiencing over the previous a number of weeks.
Whereas not the kind of data we usually share with prospects, I’m together with an in depth checklist of initiatives which might be in flight to enhance the scenario.
What we’ve already addressed
- We’ve expanded our media pipeline and transcoding capability by incorporating sources throughout a number of areas.
- We’ve made enhancements to efficiency and transactional conduct round uploads. We’ve additionally moved the asynchronous occasions generated by uploads onto a brand new, devoted job system.
- We’ve deployed enhancements to the efficiency of our socket service that powers updates and presence throughout our functions, lowering unneeded load on our infrastructure.
- We’ve made important reliability and efficiency enhancements to the archiving course of whereas upgrading it to our new job system.
- We’ve optimized database efficiency on digest jobs, which supplies consolidated notifications on your initiatives.
What’s in progress
At present we’ve a number of work streams to focus on for ongoing enchancment, and are working throughout a number of engineering disciplines concurrently. The vast majority of the efforts give attention to a number of key themes:
Database infrastructure enhancements
We’re making a major overhaul of our database infrastructure, together with enhancements and tuning to our connection pooling, multiplexing, and caching configurations to make sure we’re getting probably the most out of our knowledge layer. This may cut back load and improve the velocity to finish requests.
API efficiency
We’re presently in beta with quite a lot of efficiency enhancements to API subsystems akin to storage calculation, which reduces general load for add and archival workflows. We’re additionally upgrading different event-heavy workloads akin to asset administration operations onto our new job system, and dealing on the following revision to our exercise and bundling structure.
Increasing infrastructure capability throughout areas
We’re persevering with to increase the work we started with transferring our media pipelines throughout areas, and will likely be wanting into additional separating our infrastructure for storage, knowledge, and compute throughout areas to enhance each capability and resiliency.
Beneath these themes we presently have 9 work streams in progress, and our infrastructure and backend engineers stay centered on resolving the problems that you simply’ve skilled lately.
I will likely be actively concerned to make sure that we restore our efficiency to the extent of reliability you’ve counted on for years. Thanks on your ongoing persistence and help as we work towards reestablishing your belief in Body.io.
Leave a Reply