Operations & DevOps

Owned by Siamak Haschemi (Deactivated)
Last updated: Oct 06, 2023
3 min read

Operations & DevOps Procedures and Practices at Ingenious

Introduction

In the ever-evolving landscape of technology, Ingenious saw the necessity of moving from traditional development practices to DevOps. Before this transformation, our company grappled with infrequent, bundled releases, which made bug identification and resolution a tedious task. Moreover, the absence of automated builds and the limited use of CI/CD systems hindered our efficiency.

  • Challenge: Managing bundled releases and identifying bugs.

  • Approach: Transition to frequent releases and adoption of CI/CD systems.

“We can characterize the health of a service from the most basic requirements needed for a system to function as a service at all to the higher levels of function…”  – Site Reliability Engineering Book

DevOps Culture and Collaboration

Instead of traditional siloed roles, our team members wear multiple hats. We don’t maintain separate development and operations teams. Every developer at Ingenious bears a part of the operational responsibility, ensuring that there's a constant exchange of knowledge, fostering a culture of continuous learning and collaboration.

  • Challenge: Siloed development and operations teams.

  • Approach: Merged roles with every developer taking up operational responsibilities. Intensive use of Slack.

Continuous Integration (CI) and Continuous Deployment (CD)

Using Jenkins, orchestrated on Kubernetes, we've been able to create a seamless CI/CD pipeline. This setup supports a majority of our applications, including Java, PHP, and Typescript, thereby ensuring faster and more reliable builds. The integration of tools like JUnit and the Testcontainer library further bolsters our testing procedures.

  • Challenge: Diverse applications needing consistent CI/CD.

  • Approach: Jenkins on Kubernetes for CI/CD supporting a unified pipeline.

Infrastructure as Code (IaC)

We've progressively embraced IaC, utilizing Kubernetes yaml descriptors stored on Github. While some core components, like our Kubernetes clusters and PostgreSQL databases, were initially set up manually, we're gearing up to fully adopt IaC, ensuring enhanced resilience and preparedness for unforeseen challenges.

  • Challenge: Manual setups for core infrastructure components.

  • Approach: Adopt Kubernetes yaml descriptors and plan for a broader IaC transition.

Containerization and Orchestration

For containerization, we employ Jib for our Java applications and Dockerfiles for others. With approximately 60 deployments in our ecosystem, we've standardized deployment metadata using a centralized Jenkins-pipeline, ensuring uniformity across our infrastructure.

  • Challenge: Standardizing deployment metadata across applications.

  • Approach: Jib for Java, Dockerfiles for others, and a centralized Jenkins-pipeline.

Configuration Management

We prioritize a systematic configuration management approach. Currently, configurations are stored as yaml files in git repositories alongside their respective applications. We envision a more centralized method in the future using a mono-git-repo for all configurations, enhancing both security and manageability.

  • Challenge: Dispersed configurations affecting security and manageability.

  • Approach: Store configurations with applications and plan for a mono-git-repo.

Performance Optimization and Scalability

Backed by Prometheus for monitoring, integrated with Grafana for visualization, our infrastructure is tailored for high performance. Additionally, the Google Cloud Monitoring & Logging tools play a pivotal role in our central logging system. Our infrastructure is also optimized for scalability, utilizing both Kubernetes and Google Cloud's auto-scaling features to handle traffic surges efficiently.

  • Challenge: Efficiently handling traffic surges and maintaining performance.

  • Approach: Prometheus and Grafana integration with a two-layered auto-scaling strategy.

Feedback Loops and Continuous Improvement

Feedback is integral to our operations. Given our unified team structure, communication is streamlined, ensuring that insights from one team member benefit all. This feedback mechanism has often led to enhancements, particularly in our monitoring processes.

  • Challenge: Gaps in monitoring and continuous improvement.

  • Approach: Foster seamless communication and use feedback for refinements.

Conclusion

At Ingenious, our DevOps journey exemplifies adaptability and a forward-looking perspective. Our strategies and practices, though evolving based on current needs, are always anchored in delivering technical excellence to meet the demands of our developers and software architects.