LLM Engineer

Job Title: Senior LLM Engineer (AWS Bedrock | Agentic AI | RAG | Prompt Engineering | Vector Databases)

Location: Kolkata (On-site)

Seniority: Senior (Hands-on; expected to lead implementation & mentor team)

Team: AI Engineering / Platform Engineering

Role Type: Full-time

Role Summary

We are looking for a Senior LLM Engineer to build and productionize GenAI capabilities using AWS Bedrock.

This role involves:

• Owning prompt engineering practices
• Building RAG (Retrieval-Augmented Generation) workflows
• Working with vector databases
• Enabling model customization (fine-tuning / instruction tuning / Bedrock-supported approaches)

You will collaborate closely with Product, Backend, and Security teams to deliver reliable, scalable, and cost-efficient AI systems.

Key Responsibilities

1. Prompt Engineering & Evaluation

• Design, test, and maintain prompt templates for:
• Task decomposition
• Tool usage
• Structured outputs
• Summarization & classification
• Build prompt guardrails:
• Schema constraints
• Injection resistance patterns
• Deterministic outputs & fallback mechanisms
• Develop evaluation datasets and run:
• A/B testing
• Regression checks
• Quality scoring frameworks

2. RAG & Vector Search

• Design end-to-end RAG pipelines:
• Ingestion Chunking Embedding Indexing Retrieval Generation
• Work with vector databases such as:
• OpenSearch, Aurora pgvector, Pinecone, Weaviate, Milvus
• Improve retrieval quality using:
• Hybrid search (BM25 + vectors)
• Metadata filtering
• Re-ranking
• Caching strategies

3. AWS Bedrock Implementation

• Build and deploy Bedrock-based services:
• Model selection & routing
• Prompt orchestration
• Optimize systems for:
• Cost
• Latency
• Reliability (retry logic, fallbacks, batching, token optimization)
• Integrate with AWS ecosystem:
• IAM, KMS, CloudWatch, VPC endpoints

4. Model Customization

• Enable model customization workflows:
• Data preparation & labeling
• Safety filtering
• Fine-tuning / adapter-based methods
• Build pipelines for:
• Model versioning
• Updates & rollback
• Reduce hallucinations using:
• Grounding strategies
• Defined acceptance criteria

5. Engineering Excellence & Security

• Build secure, multi-tenant systems:
• Data isolation
• Least privilege access
• Secrets management
• Implement observability:
• Prompt-response tracing (with redaction)
• Quality metrics
• Drift monitoring
• Document:
• Architecture
• Prompt strategies
• Runbooks
• Mentor junior team members

Must Have Skills

• Strong hands-on experience in Prompt Engineering with measurable impact
• Experience building RAG systems & vector search pipelines
• Hands-on experience with AWS Bedrock
• Strong programming skills in:
• Python (preferred) / JavaScript / TypeScript / Java
• Experience with:
• LLM evaluation frameworks
• Test datasets & regression testing
• Understanding of LLM risks:
• Hallucinations
• Prompt injection
• Data leakage
• Ability to build production-grade systems (performance, reliability, cost optimization)

Good to Have

• Experience with:
• Embeddings & chunking strategies
• Hybrid retrieval & re-ranking
• Experience building:
• Agentic workflows
• Tool/function calling systems
• Knowledge of:
• Model governance
• PII handling & data privacy
• Familiarity with AWS services:
• OpenSearch, S3, Lambda, ECS/EKS, Step Functions, DynamoDB, CloudWatch
• Understanding of ML lifecycle & fine-tuning

Nice to Have

• Experience with:
• Fine-tuning open-source LLMs (e.g., LLaMA family)
• Model serving tools (vLLM, TGI)
• Familiarity with:
• Bedrock Knowledge Bases / Agents
• Experience building:
• Prompt libraries & governance workflows
• Exposure to:
• GenAI security testing (red teaming, jailbreak testing)
• Experience with:
• Multilingual NLP (Indian / European languages)

Experience & Qualification

• 510+ years of engineering experience
• 2+ years in LLM / GenAI systems (flexible for strong candidates)
• Bachelors / Masters in Engineering / Computer Science (or equivalent experience)