This text was co-authored with SK Telecom’s Seunghyun Jeong, Sunwoo Lee, and Eric Davis.
SK Telecom (SKT) is South Korea’s main telecommunications firm, serving 30 million clients and on the forefront of synthetic intelligence innovation. In keeping with SKT’s Synthetic Intelligence Pyramid technique, which goals to unleash the potential of synthetic intelligence for anybody, wherever, anytime, SKT is collaborating with the AWS Generative Synthetic Intelligence Innovation Heart (GenAIIC) customized mannequin program to discover using Amazon Bedrock to realize telecommunication-specific purposes. Area coaching mannequin use instances.
This collaboration is in keeping with SKT’s imaginative and prescient of leveraging AI experience and strategic partnerships to develop revolutionary AI-based services and products. One of many measures focuses on the event of personalized options based mostly on query and reply (Q&A) based mostly on reference paperwork.
Retrieval-augmented technology (RAG) is a well-liked method for question-answering duties that improves factual accuracy and information base. Nonetheless, RAG faces the problem of manufacturing responses that don’t match the popular tone, fashion, and strategy of the telecommunications use case, in addition to retrieving irrelevant paperwork, which can lead to inaccurate responses. To deal with this problem, SKT and AWS GenAIIC intention to make use of mannequin customization to enhance three key areas of the Anthropic Claude mannequin on Amazon Bedrock:
- Present concise and informative solutions
- Accurately cite hyperlinks in retrieved recordsdata
- Reply in a tone and magnificence in step with SKT and much like actual solutions
Moreover, the workforce explores utilizing artificial information produced by bigger giant language fashions (LLMs) to enhance the efficiency of smaller fashions for information distillation and eventualities with restricted labeled coaching information.
Amazon Bedrock is a completely managed service that gives quite a lot of LLMs and base fashions (FMs) in addition to options such because the Amazon Bedrock Data Base, Amazon Bedrock Brokers, and Amazon Bedrock Guardrails that may speed up many generative AI use instances. Amazon Bedrock is the one absolutely managed service that allows you to fine-tune Claude fashions. Amazon Bedrock supplies an intuitive and safe option to fine-tune Anthropic’s Claude fashions and extra. Nice-tuned Claude fashions may be deployed utilizing Amazon Bedrock and may seamlessly use Amazon Bedrock’s options, such because the Amazon Bedrock Data Base for telecom domain-specific RAGs or Amazon Bedrock Brokers for agent utilization.
On this article, we share how SKT used Amazon Bedrock to customise the Anthropic Claude mannequin for telecom-specific Q&A on SKT’s technical telecom paperwork.
Resolution overview
The workforce explored a mixture of on-the-fly optimization, customization (fine-tuning), and information enhancement utilizing artificial information. This multifaceted strategy goals to maximise the benefits of every method within the fundamental question-answer technology process.
Within the following sections, we discover these strategies in additional element.
Anthropic’s Claude customization and well timed optimization
Varied FMs, together with Anthropic’s Claude, may be fine-tuned via Amazon Bedrock, permitting pre-trained language fashions to be tuned for particular use instances. It is particularly efficient for customized responsive kinds and formatting to comply with.
The workforce first optimized system prompts, implementing standardized tips for reply codecs and file citations based mostly on anthropic mannequin immediate greatest practices. Key areas of focus embody:
- System directions are clearly introduced
- Use code block format constantly
- Context-based personalized responses
This mix of well timed engineering and fine-tuning resulted in important enhancements:
- ROUGE-3 rating elevated by greater than 50%
- ROUGE-L rating elevated by greater than 25%
- Embedding similarity rating elevated by greater than 4%
- Vital progress in correct reference quotation
The iterative enhancement course of demonstrated cumulative advantages, with well timed updates alone displaying a 35-40% enchancment in key indicators, and the ultimate personalized mannequin achieved a 50-60% enchancment in some indicators.
This improvement clearly illustrates the cumulative advantages of mannequin customization via RAG, dwell engineering, and fine-tuning, leading to fashions that considerably outperform baseline and dwell up to date variations by way of ROUGE scores and quotation accuracy. The ROUGE rating measures the similarity between the true worth and the generated outcome by calculating N-gram phrase overlap. The desk beneath summarizes these enhancements.
| Grasp of Legal guidelines | Well timed updates | fine-tuning | enchancment from baseline | ||
| Rouge-3 | Rouge-L | quotation accuracy | |||
| “Claude 3 Sonnets” by Anthropic | – | – | baseline | baseline | baseline |
| “Claude 3 Sonnets” by Anthropic | ✅ | – | +38.30% | +13.4% | +52.94% |
| “Claude 3 Sonnets” by Anthropic | ✅ | ✅ | +58.1% | +26.8% | +70.59% |
Complete information for fine-tuning
To deal with the problem of restricted high-quality labeled coaching information, the workforce explored artificial information technology methods. This strategy additionally facilitates information distillation from bigger LLMs to smaller, extra focused fashions, offering advantages comparable to decrease latency and value.
The workforce carried out managed experiments utilizing the next strategies:
- Baseline set of 500 actual samples
- Enhanced set of 500 authentic samples and over 1,500 artificial samples
- Bigger authentic set of two,000 samples
The synthesis was generated utilizing Anthropic’s Claude Sonnet 3, creating new question-answer pairs based mostly on the identical search recordsdata used within the floor fact examples.
Outcomes had been evaluated utilizing LL.M.-based comparative and human choice assessments. Human evaluators blindly rank mannequin outputs and assign scores based mostly on choice (greatest: 4, second: 3, third: 2, worst: 1). The desk beneath reveals the outcomes of human choice evaluation scores.
| rank | Mannequin | Cumulative rating (Very best: 160) |
| 1 | Nice-tuned utilizing 2,000 authentic samples | 114 |
| 2 | Nice-tuned utilizing 500 authentic and 1,500 artificial samples | 112 |
| 3 | Nice-tuning with 500 authentic samples | 85 |
| 4 | No fine-tuning (baseline) | 84 |
Some key findings embody:
- Small coaching set (500 samples) reveals minimal enchancment over baseline
- The bigger coaching set (2,000 samples) scores significantly higher
- Artificial enhancements carry out equally to equally sized originals
Whereas it’s all the time supreme to have a considerable amount of domain-specific coaching materials, many companies have a restricted set of supplies accessible. On this case, artificial information can play a significant position in changing the unique information. This demonstrates the potential of artificial information for mannequin customization.
in conclusion
SK Telecom’s partnership with AWS GenAIIC displays the corporate’s dedication to growing revolutionary AI options to deal with telecommunications challenges. By customizing Anthropic’s Claude mannequin utilizing Amazon Bedrock, SKT achieved important efficiency enhancements for telecom-specific Korean use instances with out having to construct a mannequin from scratch. Proof of idea demonstrated important enhancements:
- ROUGE-3 rating elevated by roughly 58%
- ROUGE-L rating elevated by about 27%
- Main enhancements in returning right reference hyperlinks
This strategy, mixed with artificial information technology know-how, aligns with SKT’s AI pyramid technique, enabling quicker testing and improvement of recent strategies. As SKT continues to deal with key areas comparable to private AI assistants, AI healthcare, and AI information facilities, the partnership with AWS represents an essential step for SKT in AI improvement and long-term competitiveness within the international AI area.
For these considering working with AWS on related tasks, please go to the Generative AI Innovation Heart.
Concerning the creator
Hong Shengmin is a senior purposes scientist within the AWS Generative AI Innovation Heart, the place he helps speed up varied use instances for AWS clients. Previous to becoming a member of Amazon, Sungmin was a postdoctoral fellow at Harvard Medical Faculty. He holds a Ph.D. PhD in pc science from New York College. Outdoors of labor, Sungmin enjoys climbing, studying and cooking.
Che Xiuzheng He’s a deep studying architect on the AWS Generative AI Innovation Heart, specializing in mannequin customization and optimization. She has in depth hands-on expertise leveraging generative AI in addition to conventional AI/ML options to unravel purchasers’ enterprise use instances. Sujeong holds a grasp’s diploma in information science from New York College.
Arijit Ghosh Chaudhry He’s a scientist on the AWS Generative AI Innovation Heart, accountable for mannequin customization and optimization. He’s dedicated to utilized analysis on fine-tuning and mannequin analysis to allow GenAI to be utilized in varied industries. He holds a grasp’s diploma in pc science from the College of Illinois at Urbana-Champaign, the place his analysis focuses on query answering, search, and area adaptation.
Cash for a month As an Utility Scientist II within the AWS Generative AI Innovation Heart, she helps the supply of generative AI options to AWS clients. On this position, she works with a workforce of specialists to develop revolutionary AI-driven fashions for AWS clients throughout industries. Yiyue holds a Ph.D. She holds a PhD in pc science from the College of Notre Dame, the place her analysis focuses on superior machine studying and deep studying methods.
Chen Weizhi He’s a machine studying engineer on the AWS Generative AI Innovation Heart, accountable for mannequin customization and optimization of the LL.M. He additionally builds instruments to assist his workforce tackle all facets of the LLM improvement lifecycle, together with fine-tuning, benchmarking, and cargo testing, thereby accelerating AWS buyer adoption for quite a lot of use instances. He holds a grasp’s diploma in pc science from the College of California, Davis.
Hannah Marlowe He’s a senior supervisor of mannequin customization on the AWS Generative AI Innovation Heart. Her workforce focuses on serving to purchasers leverage their distinctive proprietary information to develop differentiated generative AI options to realize important enterprise outcomes. She holds a PhD in physics from the College of Iowa, specializing in astronomical X-ray evaluation and instrument improvement. When not working, she spends her time climbing, mountain biking, and snowboarding within the mountains of Colorado.
Jung Seung Hyun (Steve) He’s the workforce chief of the SKT platform utility workforce. He’s accountable for the commercialization of the World Intelligence Platform (GIP), which supplies synthetic intelligence fashions and instruments. For many of his profession, he has been a mission supervisor growing varied cellular companies for SK, comparable to cellular pockets, trend streaming and unified login companies. His workforce is increasing the supply of fashions and options to make it simpler for inner groups to use AI, contributing to SKT’s AI transformation. Earlier than getting into the sector of synthetic intelligence, he was a product supervisor, growing and working varied cellular companies, comparable to cellular wallets, trend streaming media, and unified login companies in america and South Korea.
Lee Solar Woo(Lois) He’s the workforce chief of the information building and analysis workforce of SK Telecom’s international synthetic intelligence know-how division. She oversees the design and building of language mannequin coaching supplies, the mannequin efficiency analysis course of, and its utility in companies. Her profession has been targeted on NLP within the IT area, which inserts effectively together with her background in linguistics and Korean language training. Along with a world-class workforce, she continues to discover and resolve fascinating issues, comparable to the right way to optimize the design of supplies for language mannequin coaching, duties and strategies to confirm the efficiency of language fashions, and the optimum design of synthetic intelligence to speak to people.
Eric Davis He’s the Vice President of SKT’s Synthetic Intelligence Know-how Collaboration Group. Eric is accountable for technical cooperation with international know-how companions to customise giant language fashions (LLM) for the telecommunications area. His workforce is accountable for designing and developing datasets to tune the LLM, in addition to benchmarking the LLM generally and the LLM in telecommunications. Eric holds a grasp’s diploma in pc science from Carnegie Mellon College’s Faculty of Language Know-how and a bachelor’s diploma in linguistics and psychology from UCLA.