{"id":75169,"date":"2024-12-01T16:23:36","date_gmt":"2024-12-01T16:23:36","guid":{"rendered":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/"},"modified":"2024-12-01T16:23:36","modified_gmt":"2024-12-01T16:23:36","slug":"introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode","status":"publish","type":"post","link":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/","title":{"rendered":"Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode"},"content":{"rendered":"<div class=\"youtubomatic-video-container\"><iframe loading=\"lazy\" width=\"580\" height=\"380\" src=\"https:\/\/www.youtube.com\/embed\/7h4G53etywI?autoplay=1&#038;controls=1&#038;hl=en\" frameborder=\"0\" allowfullscreen><\/iframe><\/div>\n<p>(Emily Tour, Kabir Manandhar Shrestha) The last five years have seen a significant increase in the application of machine learning to the study of ancient scripts. Applications are broad, and include recognition via Optical Character Recognition (OCR), textual restoration, palaeographic analysis, topic modelling, representation learning, decipherment and machine translation (Sommerschield 2023). A large number of ancient language corpora have been digitised in recent decades, supporting this research. However, while the necessary Unicode blocks for many of these ancient scripts are available, a number of these data sets are still presented as Romanised transliterations.<\/p>\n<p>In response to this situation, we have created Potnia (<a href=\"https:\/\/pypi.org\/project\/potnia\/)\" target=\"_blank\">https:\/\/pypi.org\/project\/potnia\/)<\/a>, an open-source Python language library under the Apache 2.0 license, designed to convert such transliterated texts to Unicode. The session image accompanying this proposal provides an example of Potnia\u2019s conversion process, with a Romanised transliteration of a Linear B text as the input, and the Unicode representation of this same text as the output. This conversion is crucial for downstream machine learning tasks, as tokenisation in the original Unicode script allows for more accurate representation of linguistic structures and mitigates potential biases introduced by transliteration.<\/p>\n<p>Potnia&#8217;s flexible architecture, built on Python&#8217;s object-oriented principles, employs string manipulation techniques and regular expressions to handle various complexities inherent in ancient texts, such as uncertain readings, missing elements, and script-specific notations. At present, the library can be used for Linear B texts, with functionality for Linear A, Sumerian and Akkadian soon to follow.<\/p>\n<p>Potnia&#8217;s design allows for easy addition of new scripts, each with its own set of rules for tokenisation, regularisation, and character mapping. This extensibility positions us well for future inclusion of additional scripts. To ensure reliability and facilitate open-source contributions, we&#8217;ve implemented a comprehensive test suite using pytest, with test cases defined in YAML files for easy expansion. This approach covers key functionalities across different scripts and simplifies the process of adding new test scenarios as the library grows.<\/p>\n<p>\u2039b\u203a\u2039em\u203aReferences\u2039\/em\u203a\u2039\/b\u203a<br \/>\nSommerschield, T., Y. Assael, J. Pavlopoulos, V. Stefanak, A. Senior, C. Dyer, J. Bodel, J. Prag, I. Androutsopoulos, and N.D. Freitas. 2023. \u201cMachine Learning for Ancient Languages: A Survey.\u201d \u2039em\u203aComputational Linguistics\u2039\/em\u203a 49 (3): 1\u201345. doi: 10.1162\/coli_a_00481.<\/p>\n<p><a href=\"https:\/\/pretalx.com\/pycon-au-2024\/talk\/HMWPGH\/\" target=\"_blank\">https:\/\/pretalx.com\/pycon-au-2024\/talk\/HMWPGH\/<\/a><\/p>\n<p>python, pycon, australia, programming, conference, technical, developers, panel, sessions, libraries, frameworks, community, sysadmins, students, education, data, science<\/p>\n<p>Videos licensed as CC-BY-NC-SA 4.0<\/p>\n<p>PyCon AU is the national conference for the Python programming community, bringing together professional, student and enthusiast developers, sysadmins and operations folk, students, educators, scientists, statisticians, and many others besides, all with a love for working with Python.<\/p>\n<p>Licensed as CC BY-NC-SA &#8211; <a href=\"http:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/\" target=\"_blank\">http:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/<\/a><\/p>\n<p>Produced by Next Day Video Australia: <a href=\"https:\/\/nextdayvideo.com.au\" target=\"_blank\">https:\/\/nextdayvideo.com.au<\/a><\/p>\n<p>Sat Nov 23 13: 30: 00 2024 at Eureka 2<\/p>\n","protected":false},"excerpt":{"rendered":"<p>(Emily Tour, Kabir Manandhar Shrestha) The last five years have seen a significant increase in the application of machine learning to the study of ancient scripts. Applications are broad, and include recognition via Optical Character Recognition (OCR), textual restoration, palaeographic analysis, topic modelling, representation learning, decipherment and machine translation (Sommerschield 2023). A large number of ancient language corpora have been digitised in recent decades, supporting this research. However, while the&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","tve_updated_post":"","tve_custom_css":"","tve_user_custom_css":"","tve_globals":{},"tcb2_ready":0,"tcb_editor_enabled":0,"tve_landing_page":"","_tve_header":"","_tve_footer":""},"categories":[1],"tags":[],"class_list":["post-75169","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode - UshopWell.com<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode - UshopWell.com\" \/>\n<meta property=\"og:description\" content=\"(Emily Tour, Kabir Manandhar Shrestha) The last five years have seen a significant increase in the application of machine learning to the study of ancient scripts. Applications are broad, and include recognition via Optical Character Recognition (OCR), textual restoration, palaeographic analysis, topic modelling, representation learning, decipherment and machine translation (Sommerschield 2023). A large number of ancient language corpora have been digitised in recent decades, supporting this research. However, while the...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/\" \/>\n<meta property=\"og:site_name\" content=\"UshopWell.com\" \/>\n<meta property=\"article:published_time\" content=\"2024-12-01T16:23:36+00:00\" \/>\n<meta name=\"author\" content=\"UShopWell\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"UShopWell\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\\\/\"},\"author\":{\"name\":\"UShopWell\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#\\\/schema\\\/person\\\/6fd1f9e0ff932e534c86c70d5acff0fc\"},\"headline\":\"Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode\",\"datePublished\":\"2024-12-01T16:23:36+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\\\/\"},\"wordCount\":468,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#organization\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\\\/\",\"url\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\\\/\",\"name\":\"Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode - UshopWell.com\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#website\"},\"datePublished\":\"2024-12-01T16:23:36+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#website\",\"url\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/\",\"name\":\"UshopWell.com\",\"description\":\"The Premiere Online Marketplace\",\"publisher\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#organization\",\"name\":\"UshopWell\",\"url\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/wp-content\\\/uploads\\\/2018\\\/01\\\/pandaSwea.png\",\"contentUrl\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/wp-content\\\/uploads\\\/2018\\\/01\\\/pandaSwea.png\",\"width\":365,\"height\":359,\"caption\":\"UshopWell\"},\"image\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#\\\/schema\\\/person\\\/6fd1f9e0ff932e534c86c70d5acff0fc\",\"name\":\"UShopWell\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g\",\"caption\":\"UShopWell\"},\"url\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/author\\\/kburnettu\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode - UshopWell.com","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/","og_locale":"en_US","og_type":"article","og_title":"Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode - UshopWell.com","og_description":"(Emily Tour, Kabir Manandhar Shrestha) The last five years have seen a significant increase in the application of machine learning to the study of ancient scripts. Applications are broad, and include recognition via Optical Character Recognition (OCR), textual restoration, palaeographic analysis, topic modelling, representation learning, decipherment and machine translation (Sommerschield 2023). A large number of ancient language corpora have been digitised in recent decades, supporting this research. However, while the...","og_url":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/","og_site_name":"UshopWell.com","article_published_time":"2024-12-01T16:23:36+00:00","author":"UShopWell","twitter_card":"summary_large_image","twitter_misc":{"Written by":"UShopWell","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/#article","isPartOf":{"@id":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/"},"author":{"name":"UShopWell","@id":"https:\/\/ushopwell.com\/ublog\/#\/schema\/person\/6fd1f9e0ff932e534c86c70d5acff0fc"},"headline":"Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode","datePublished":"2024-12-01T16:23:36+00:00","mainEntityOfPage":{"@id":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/"},"wordCount":468,"commentCount":0,"publisher":{"@id":"https:\/\/ushopwell.com\/ublog\/#organization"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/","url":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/","name":"Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode - UshopWell.com","isPartOf":{"@id":"https:\/\/ushopwell.com\/ublog\/#website"},"datePublished":"2024-12-01T16:23:36+00:00","breadcrumb":{"@id":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ushopwell.com\/ublog\/introducing-potnia-a-python-language-library-for-the-conversion-of-ancient-texts-to-unicode\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ushopwell.com\/ublog\/"},{"@type":"ListItem","position":2,"name":"Introducing Potnia: A Python language library for the conversion of ancient texts to Unicode"}]},{"@type":"WebSite","@id":"https:\/\/ushopwell.com\/ublog\/#website","url":"https:\/\/ushopwell.com\/ublog\/","name":"UshopWell.com","description":"The Premiere Online Marketplace","publisher":{"@id":"https:\/\/ushopwell.com\/ublog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ushopwell.com\/ublog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ushopwell.com\/ublog\/#organization","name":"UshopWell","url":"https:\/\/ushopwell.com\/ublog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ushopwell.com\/ublog\/#\/schema\/logo\/image\/","url":"https:\/\/ushopwell.com\/ublog\/wp-content\/uploads\/2018\/01\/pandaSwea.png","contentUrl":"https:\/\/ushopwell.com\/ublog\/wp-content\/uploads\/2018\/01\/pandaSwea.png","width":365,"height":359,"caption":"UshopWell"},"image":{"@id":"https:\/\/ushopwell.com\/ublog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ushopwell.com\/ublog\/#\/schema\/person\/6fd1f9e0ff932e534c86c70d5acff0fc","name":"UShopWell","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g","caption":"UShopWell"},"url":"https:\/\/ushopwell.com\/ublog\/author\/kburnettu\/"}]}},"_links":{"self":[{"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/posts\/75169","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/comments?post=75169"}],"version-history":[{"count":0,"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/posts\/75169\/revisions"}],"wp:attachment":[{"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/media?parent=75169"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/categories?post=75169"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/tags?post=75169"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}