· Joseph · Today I Learn  · 3 min read

[TIL] New service sharing: Hex.tech

Podcast source: Hex: Data Project Sharing with Caitlin Colgrove and Barry McCardel

最近迷上聽podcast,裡面專門介紹新服務跟技術,同時可以練練聽力跟學習新知,既然學了乾脆把它寫成心得分享出來。今天來介紹一個新聽到的服務: Hex,在解決大數據專案資料共享的問題。

Hex screenshot

雖然我不是個資料工程師,但曾經使用過Python Jupyter Notebook,而且中間又經歷過換電腦,要重跑都得安裝一堆有的沒的,每次用起來都很不方便。但這次聽到這個服務也是讓我滿臉問號,Jupyter不是已經很多人用了?為什麼又有一個類似的服務出來?

這次訪問到的兩個人Caitlin跟Barry是Palantir的同事,Palantir共同創辦人是Paypal的Peter,他們的客戶都是赫赫有名的大公司大企業或政府機關,但Caitlin跟Barry在Palantir裡看到客戶們導入Data-driven時的種種困難,才衍伸出這個Hex產品。當資料越來越多,就出現共享資料的問題;當團隊越來越大,就出現共同討論的問題。更宏觀一點來看,其實Hex是為了解決整個workflow的問題,會不會我在測試的方法別人用過了?會不會我用同樣的方法可以但你用同樣的方法不行?每次溝通都要透過切換軟體?討論結果還得匯出成PDF傳給對方?Hex可以讓data team的workflow又可以解決討論跟共享的問題,這很像真的挺有價值的。

中間訪問到技術面如何解決共享問題時,Caitlin提到了一個Conflict-free replicated data type(CRDT)演算法,還有另一套Operational Transformation(OT),這兩個都是他們在處理real-time功能時的tradeoff。再加上他們優化了Apollo跟GraphQL,最後才達到他們想要的協作體驗。

這篇podcast還提到了很多很多資料科學領域會用到的tools跟一些BI的工具,甚至是新的data pipeline,而這些也是現在資料科學家要一直學習的東西。或許未來pipeline會不一樣,但他們應該也會更快整合並讓大家使用。

Back to Blog

Related Posts

View All Posts »
Switching from vim to Neovim!

Switching from vim to Neovim!

I've been using Vim for a long time and finally switched to Neovim. The initial thought of switching came after the author of VIM passed away in August 2023, as I didn’t have the time to try other editors. After a year, “vibe coding” grew up, so I started thinking how to integrate AI into my editor and surveying how to use AI in Vim, which led to this journey. TOC Main differences Tab vs. Buffer: ref: https://www.reddit.com/r/neovim/comments/13iy0p0/why_some_people_use_buffers_for_tabs_and_not_vim/ In vim, I used vsp or tabnew to open or edit a file and mapped tabprevious command with Tab key to navigate between tabs. However, in Neovim, I used buffer instead of tabs and mapped BufferLineCyclePrev with Shift + h for switching buffers. coc vs. native lsp: I configured many coc settings to support TypeScript and JavaScript language servers, including linting and Prettier on save, go-to definition or reference, and codelens. After using Neovim, I converted all of these settings to nvim-lspconfig and mason, among others. Lua supports: Although I’m not familiar with Lua, it allows me to write more readable configuration files using modules, functions, and tables (objects) for Neovim. AI supports: Vibe coding! I found many plugins to integrate LLMs and finally selected three to use: avante.nvim Let your Neovim be a Cursor AI IDE – it's undergoing rapid iterations and many exciting features will be added successively; it updates almost every day! CodeCompanion CodeCompanion can customize workflows and prompts, and its Action Palette is a really useful tool. gp.nvim GP means GPT Prompt, and it's an Neovim AI plugin. It helps me write prompts through Neovim's cmdline. I usually switch between these plugins, and I'm still thinking about my 'vibe-way' to use them. Because of supporting Ollama LLM, all of them can be used offline. For now, I've used Neovim for three months, and got to know Neovim. In my experience, Neovim is really more productive than vim. Reference AI in Neovim How to set up Neovim for coding React ZazenCodes Neovim Rewrite by Avante + Ollama gemma3:4b ------ For a long time, I relied on Vim for my coding. However, after Bram Moolenaar’s passing in August 2023 – a significant influence in the Vim community – I decided it was time for a change. I began exploring alternative editors, ultimately settling on Neovim. This journey wasn’t just about switching editors; it was about integrating AI into my workflow and redefining my coding experience. Key Differences: Tabs vs. Buffers One of the first things I noticed was the shift from tabs to buffers in Neovim. In Vim, I frequently used vsp or tabnew to open or edit files, navigating between tabs with the Tab key and Tabprevious command. Neovim, however, utilizes buffers, offering a more streamlined approach. I configured BufferLineCyclePrev with Shift + h for seamless buffer switching, alongside nvim-tree/nvim-web-devicons and akinsho/bufferline.nvim. Leveraging Language Servers with coc and nvim-lspconfig I configured many coc settings to support TypeScript and JavaScript language servers, including linting and Prettier on save, go-to definition or reference, and codelens. Recognizing the power of language servers, I then converted all of these settings to nvim-lspconfig and mason.nvim, streamlining my development environment. Lua Configuration for Readability Although I’m relatively new to Lua, it allows me to write more readable configuration files using modules, functions, and tables (objects) for Neovim. Here’s a snippet of my configuration: AI-Powered Productivity with avante.nvim, CodeCompanion, and gp.nvim To truly elevate my coding experience, I integrated several AI plugins. I selected: avante.nvim**: This plugin transforms Neovim into a Cursor AI IDE, undergoing rapid iterations and adding exciting new features daily. CodeCompanion**: This plugin allows for customizable workflows and prompts, with a particularly useful Action Palette. gp.nvim**: (GPT Prompt) – This plugin helps me write prompts through Neovim’s command line interface, leveraging folke/noice.nvim. Because of supporting Ollama LLM, all of these plugins can be used offline. I’m still experimenting with how to best utilize these plugins – a “vibe-way” to coding! Resources for Further Exploration AI in Neovim How to set up Neovim for coding React ZazenCodes Neovim

AI code review with n8n

AI code review with n8n

Previously I read a post "Automate and Accelerate GitLab Code Reviews with OpenAI and n8n.io". This made me wonder: If I don’t choose GitHub Copilot for code reviews, can I still integrate AI and n8n with GitHub PR reviews? I haven’t written a blog in a long time—it’s time to start again!

2024 year in review

終於把我的部落格改成Astro了...雖然這是我2024的代辦事項。遷移完以後就要開始來補文章了。好,雖然有點晚,但還是來review一下2024: 開始了一個新的AI side project,選擇用第一次接觸的Python Fastapi + Nextjs實作,前期也真是熬了一陣子,不過是真的滿有意思的 幫客戶上線了一個新的服務,同時也不落掉維護、更新既有產品線 從2024開始把部落個從Hexo改寫成Astro,後來忙(懶)了一陣子,一直到最近才完成 學會用n8n來處理一些自動化任務,同時用它來為我的github PR做code review。為什麼不直接串Github Copilot? 因為不用錢啊 從10月開始我每週有更多時間了,也因此可以找更多Part time or case來做。 完成了一次家庭旅遊,放下工作全家人一起去大阪跟京都玩。 再來2025應該會是個更忙碌的一年,過個農曆年先來好好調適一下! Finally, I’ve migrated my blog to Astro… although this was one of the tasks for 2024. After finishing the migration, I need to start writing more blogs on it. Good, although it’s a little late, I still want to review my 2024: I started a new AI side project, choosing to implement it with Python FastAPI + Next.js, which was a pretty long and tiring period at the beginning, but it was really interesting. I launched a new service for a client, while also maintaining and updating existing product lines. I started migrating my blog from Hexo to Astro, although I finished it in 2025. I learned how to use n8n to handle some automation tasks, and I used it to implement AI code review for my GitHub pull requests (because it’s free!). Since October, I’ve had more time each week, so I can look for more part-time or case work. I completed a family trip to Osaka and Kyoto! Looking ahead to 2025, it’s going to be a much busier year! Let’s take a good rest before the Lunar New Year!

Google Ads搜尋關鍵字廣告設定

Google Ads搜尋關鍵字廣告設定 當我們有需要被解決的需求,像牙齒痛,想知道哪裡有診所可以治療? 要買電視,想知道電視的商品資訊及評價?多數人會使用Google搜尋引擎搜尋找解答,用戶搜尋意圖明確,因此在Google搜尋引擎可以接觸到精準的潛在顧客,已是電商品牌行銷布局必要的媒體工具之一。 想投放Google搜尋關鍵字廣告卻不知道要怎麼開始?本篇整理我投放Google關鍵字廣告設定過程,現在就來製作第一則廣告吧~ 投放廣告前準備事項 1. 訂好廣告目標 業主想提升網站流量,投放以流量為主的搜尋廣告及點擊出價做設定, 需依照廣告目標,選擇適合的廣告類型及出價策略,才能提升廣告投放的效益!