Hello world!

I’m a Senior Engineering Leader with deep hands-on expertise in designing and scaling distributed systems, data platforms, and cloud-native architectures. Currently, I work as a Senior Architect at Josys, where I’m also a founding engineer—helping build the core platform, engineering foundations, and architectural direction as the company scaled from early-stage to a mature enterprise SaaS.

Over the last 13+ years, I’ve worked across telecom, legal, and enterprise SaaS domains, operating at the intersection of technology, product, and business. My experience goes beyond system design—I’ve been closely involved in shaping engineering practices, platform strategy, and scale-readiness decisions that enable teams to move fast without compromising reliability or long-term growth.

I enjoy solving complex problems that sit at the edge of scale and ambiguity: architecting resilient distributed systems, building data and search platforms, optimizing cost and performance, and mentoring teams to think in systems rather than features. This site is a snapshot of my journey—what I’ve built, how I think, and the principles I apply when designing technology and teams meant to scale.

Writings

  1. Engineering Decisions behind introducing transactions in AWS DynamoDB: https://medium.com/@vipulbest1/engineering-decisions-behind-introducing-transactions-in-aws-dynamodb-3466a89af4e5

  2. Understanding NodeJs Event Loops, Phases and beyond…(Part1): https://medium.com/@vipulbest1/understanding-nodejs-event-loops-phases-and-beyond-part1-e87f40d0e3c6

  3. Handling Concurrency through database in rails: Part 2 : https://medium.com/@vipulbest1/handling-concurrency-through-database-in-rails-part-2-136e5420bce0

  4. Handling Concurrency through database in rails: Part 1: https://medium.com/@vipulbest1/handling-concurrency-through-database-in-rails-part-1-8619ffe75f2d

  5. Distributed Locks Using Ruby and Redis: https://medium.com/@vipulbest1/distributed-locks-using-ruby-and-redis-2e0a2c7b9765

  6. Thread Safety in Ruby and the mystical GIL: https://medium.com/@vipulbest1/thread-safety-in-ruby-and-the-gil-884b6bc47219

  7. ElasticSearch: Understanding Nested Data Types: https://medium.com/@vipulbest1/elasticsearch-understanding-nested-data-types-8f523364e07

Engineering Beyond the Code: What the world needs!

  1. Mysql/AWS RDS : MyIASM and InnoDB Engine

  2. MongoDB and AWS DocumentDB

  3. ElasticSearch and AWS Opensearch

  4. AWS RDS with Scalable PostgreSQL,

  5. Familiarity with YugaByteDB (Infinitely Scalable Postgress)

  6. AWS DynamoDB

  7. Index management ranging from RDBMS to documentStores :
    B+trees, secondaryIndexes, 2DSphere Indexes, Inverted Trees

  8. High Write Dbs: Spanner, RocksDB, Ubers M3DB

  9. ACID, Data consistency, Sharding, Replication strategies

  10. Transactional Guarantees in highly scalable distributed document store.

  1. Ruby and Redis based job framework: sidekiq, sidekiq-unique, rescue , delayed-job

  2. Integrated Shoryuken: Multi threaded Ruby SQS Poller

  3. Node and Redis based job framework: BullMQ

  4. AWS SQS, AWS SNS, AWS EventBridge

  5. Queue BackPressure and concurrency control in low latency , high throughput NodeJS systems using p-limit, async-queue and p-queue

  1. Parquet: Columnar File Formats

  2. Delta Lake, Iceberg TableFormats on top of parquet to provide snapshot isolations, MVCC, transactions and much more

  3. Architected high load pipelines with Apache Spark

  4. CDC Pipelines using debezium, Kafka and AirByte

  5. Architected multiple systems using medallion architecture

  6. Data Pipeline Orchestration on Apache Airflow and AWS MWAA. Scaling thousands of DAGs for consistent performance

    and latency. Architected the entire DAG pipeline for scheduled and high priority DAGs

  7. Streaming Pipelines using Spark Streaming and Flink

  8. Architected and guided team on Apache Pinot integration for low latency user queries.

  1. Microservice orchestration and SAGA using Temporal

  2. Deployed Temporal on EKS.

  3. Data Engineering workflows using AirFlow DAGs.(AWS MWAA)

Producers

  1. Compressing streaming payloads using zstd, gzip and snappy

  2. Batching and fine tuning payload size and format write consensus and quorom strategy

  3. Atleast , Atmost and Exactly once publish semantics

  4. Idempotent Publish of events

  5. Handling InSyncReplicas and ACKs

  6. De-Duplication of published events

  7. Manual and auto commit of offsets

  8. Transactional Guarantees

  9. Order of publish

  10. Partitioning strategies

Consumers

  1. Key decisions on order of consumptions

  2. Designing idempotent consumer

  3. De-duplication on consumers

  4. TimeOuts and Retries

  5. Distributed locking for duplicate consumption

  6. Out of order consumption of stale events

  7. Strict consumption order when there is dependency between two events

  8. Offset Management : Log End and committed offset, Consumer Lag monitoring and alerts, FineTuning consumer groups

  9. Partition assignment strategies

  1. Developed , deployed and maintained search systems with Read QPS of more than 1000-2000 RPS.

  2. Handle different tokenization approach for different languages

  3. Text based searches, Faceted and fuzzy searches

  4. Taking critical decisions on index model design to faster searches. Designing efficient sharding strategy

  5. Designing efficient replication strategy

  6. Optimistic Concurrency Control and handling stale events

  7. Handling Event Storms for CQRS architecture

  8. TradeOffs between Full Immutable writes vs Partial mutable writes

  9. Handling deep pagination

  1. Databases: Aurora, documentDB, Opensearch DynamoDB, AWS TimeStreams

  2. CI/CD: AWS CodeBuild and CodePipeline

  3. Queues: Sqs, Sns, Event Bridge

  4. Storage: EBS, S3

  5. S3 Federated Query: Athena

  6. Infrastructure: AWS ECS, EKS, EMR

  7. Auto scaling policies, Cloudwatch logs, ALB grouping,

  8. AWS Cloud Map, ECS Discovery Service using AWS Cloud Map, AWS Parameter store, AWS KMS