四兩撥千斤！ 創新工場首席科學家AI大牛周明博士率瀾舟團隊刷新CLUE新紀錄，輕量化模型孟子一鳴驚人！

本週，中文語言理解權威評測基準CLUE榜單，被「低調」刷新。

不同的是，不是大公司、不是超大模型……

一個新面孔，一個輕量化模型，首戰即登頂，四兩撥千斤。

CLUE榜單近年來由巨頭——騰訊、搜狗、華為、阿里達摩院輪番霸榜的格局，被首次打破。

瀾舟科技-創新工場推出的孟子模型，以十億參數完成了此前百億、千億參數模型刷新的紀錄。

這也是瀾舟科技首次對外曝光，背後團隊負責人，正是創新工場首席科學家、全球AI大牛周明博士。以下文章解釋了這個模型的原理，文章來自《量子位》微信公眾號，經授權轉載。

▎輕量化模型孟子？

孟子，基於瀾舟團隊自研技術研發的大規模預訓練語言模型。

包括創新工場、上海交通大學、北京理工大學等單位參與聯合研發。

可處理多語言、多模態數據，同時支持多種文本理解和文本生成任務，能快速滿足不同領域、不同應用場景的需求。

孟子模型基於Transformer架構，僅包含十億參數量，基於數百G級別涵蓋互聯網網頁、社區、新聞、電子商務、金融等領域的高質量語料訓練。

但誰也沒想到，小模型卻有大智慧，一經登場，打破格局。

CLUE，中文語言理解領域最具權威性的測評基準，涵蓋文本相似度、分類、自然語言推理、閱讀理解等共10項語義分析和理解類子任務。

該榜單競爭激烈，幾乎是業內所有自然語言理解玩家必爭之地。

騰訊、搜狗、華為、阿里達摩院等更是輪番霸榜刷新紀錄。

而且隨著大參數模型愈演愈烈，CLUE還漸有巨頭壟斷之勢。

因為百億、千億甚至萬億參數的大模型，已然不再是創業或其他玩家可與之爭鋒。

萬萬沒想到，瀾舟科技-創新工場團隊出手，四兩撥千斤。

因為孟子，走的是基於輕量級、高效訓練的研究路線，致力於構建十億級別的小模型，充分發揮已有參數下的模型潛力，有利於快速、低成本地落地現實業務場景。

孟子預訓練模型性能比肩甚至超越千億大模型，在包含文本分類、閱讀理解等各類任務上表現出色。

相對已有的中文語言模型，孟子模型實現了多項突破性進展：
 1） 堅持「小而精」的輕量化訓練策略。實現在同等模型規模下，遠超公開模型的性能。作為精巧的小模型，對標「巨無霸」，小模型性能超越千億規模模型。
 2）使用知識圖譜增強模型，讓 AI 真正獲得知識。孟子模型具備頂尖的語言理解能力，在權威CLUE中文理解評測的總排行榜，以及分類排行榜和閱讀理解排行榜均位列第一，刷新三項榜單世界紀錄。總排行榜分數突破84分，逼近人類基準分數（85.61）。
 3）靈活的領域和場景適應能力，方便快速定制和應用。基於T5-style的端到端生成的訓練範式，同步適配BERT-style的判定式架構，既能理解也能生成。便於適配行業應用，覆蓋廣泛業務場景。

當然，隨著孟子一鳴驚人，也必然能讓輕量化模型研究來到聚光燈下。

▎原理方法和應用？

在輕量化模型算法研究方面，基於自研的基於語言學知識、知識圖譜和領域數據增強等技術，從模型架構（包括基礎層Embedding表示和交互層Attention機制）到預訓練策略進行了全方位改進。

具體有四方面：
 1) 模型結構方面，將語義角色、詞性標註等語言學特徵融合到Embedding表示中，基於句法約束引入註意力機制中，從而提升模型對語言學知識的建模能力。
 2) 訓練策略上，引入基於實體知識和Discourse的Mask機制，強化模型對語言成分和語篇關係的表徵。
 3) 為進一步提高訓練效率，使用了大模型蒸餾和初始化小模型策略。
 4) 為更好地將孟子模型適應垂直領域如金融、營銷，使用了領域數據繼續訓練並構造相應的提示模版（Prompt），取得了明顯的性能提升。

基於以上算法策略，實現從語料中高效學習涵蓋詞級、句子級和語篇級知識，大幅提升語言模型提煉語言結構和語義信息能力，以及良好的領域遷移能力，適應廣泛的產品應用場景。

另外，在Finetune的進展方面，如何將預訓練模型用於各項任務？

瀾舟團隊也有總結，從數據增強、知識蒸餾、遷移訓練、訓練優化等方面展開了一些探索，進一步提升語言模型的性能：

 1) 數據增強：使用領域相關數據；
 2) 知識蒸餾：基於Teacher-Student自蒸餾提升訓練效率；
 3) 遷移訓練：結合課程學習的思想，由易到難訓練下游模型；
 4) 訓練優化：使用多種訓練目標，多角度提升模型能力；

而且孟子還已經展開了垂直化領域應用。

基於領域適應技術，孟子模型已深度垂直化賦能相應行業。典型的例子為適用於金融領域的孟子模型，領域適應策略主要包含兩大方面：

 1) 通過大規模的泛金融領域語料，將通用孟子模型遷移到金融領域。金融版孟子模型已經應用於多個金融行業的合作企業，在金融知識圖譜搭建、脫水研報、公告抽取等多個任務上獲得了出色的表現。
 2) 通過大規模的營銷領域語料，將孟子模型遷移到數字營銷領域，完成了營銷文案生成、新聞摘要等多項任務，將用於行業頭部的數字營銷公司和多個世界五百強企業的合作之中。

瀾舟方面還透露，孟子模型已在多個領域成功落地實踐，衍生出多項行業領先的產品，涵蓋文本生成、行業搜索、機器翻譯等諸多領域。

並且毫無疑問的是，因為輕量級模型具有的模型參數較少、快速推斷的特點，更易於線上部署和推廣到移動設備中，自然不會局限於現有應用和場景，接下來還會有更廣泛的研究和應用場景中。

▎瀾舟團隊？

最後，也簡單介紹本次一鳴驚人的新面孔瀾舟科技。

瀾舟科技是創新工場孵化的一家認知智能公司。公司創始人——周明博士。

AI領域內，周明已不用過多介紹，他是公認的世界級AI科學家，自然語言處理領域的代表性人物。

周明博士在2020年加盟創新工場，擔任創新工場首席科學家。
而瀾舟科技則針對商業場景的數字化轉型，基於大數據、知識圖譜和行業模型，提供新一代的信息檢索、知識推理和商業洞見技術和相關產品。

據稱目前已與國內外幾十所著名高校和十餘個相關領域的頭部企業建立了穩定的合作關係。

值得注意的是，瀾舟科技除了大牛坐鎮，其實也是行業趨勢的體現。

引用創新工場董事長兼CEO李開復最新分享來說：

AI的發展可以按照兩個時間點劃分。

第一個時間點是2015年，以CNN為核心的計算機視覺技術讓機器超越了人類，帶來了人臉識別、智能質檢、無人零售、智慧城市、無人駕駛等商機。

而第二個時間點出現在2019年，以大模型為代表的自然語言方向取得突破性進展，讓NLP從數據、信息走向知識和洞見成為可能，將會在翻譯、語音識別、法律、金融、新聞、廣告、醫療、娛樂等大賽道帶來機遇。

「如果說CNN造就了今天計算機視覺領域的突破和眾多應用，預訓練大模型+微調也將帶來自然語言的百花齊放的發展，用數據智能驅動各類業務的升級。瀾舟科技在周明老師的帶領下取得了今天的成果，在新機遇面前躬身入局，一起發掘NLP領域的黃金發展期」，李開復說到。

Eli Clifton 挖挖挖。

錢是台灣人民納稅，怎麼用卻不被揭露。美國智庫是拿錢辦事，還是以金援為名收受獻金為實？

-----

以下中文新聞內容轉自 旺報：

https://www.chinatimes.com/newspapers/20200619000151-260309?chdtv

華府智庫昆西治安研究所（Quincy Institute）民主外交政策計畫執行人克里夫頓（Eli Clifton）於17日發表《台灣金援智庫：無所不在但很少揭露》（Taiwan funding of think tanks: Omnipresent and rarely disclosed）一文指出，台灣金援了美國五大智庫，促使這些智庫向美國執政者建言，做出有利於台灣綠營執政政府的美國政策。該文發出後，台北經濟文化代表處政治組組長趙怡翔緊急在推特上指出，該文有錯失之處，且並未向台北經濟文化代表處查證置評。

文章稱，台灣金援的五個華府智庫包括布魯金斯學會、美國進步中心（CAP）、新美國安全中心（CNAS）、戰略與國際研究中心（CSIS）以及哈德遜研究所，都有來自於台北經濟文化代表處的資金，但都將其深埋在其年度報告中。這五個華盛頓最著名、看似公正的的智庫一直在發表政策文件，敦促美國與台灣建立更緊密的關係，擴大美國與台灣的武器銷售和貿易協定，卻沒有廣泛披露其背後來自台北經濟文化代表處（TECRO）的高額資金。

支持台有助民主自由

文章詳列台北經濟文化代表處對五大智庫的贊助金額，及這些智庫收到贊助後為台灣做了什麼。文章稱，布魯金斯學會學者於2019年12月為《台北時報》撰文，指出美國兩黨支持在台灣和美國的重要性；美國進步中心研究員在今年3月分《華盛頓月刊》上發表專欄文章稱，加強美台關係將有助民主自由，並在2019年9月發表《如何支持亞洲的民主與人權》報告時，直接向美國決策者「為如何穩固支持台灣」提供直接建議。

而新美國安全中心向華盛頓提供有關2020年《中國崛起的挑戰》報告時，敦促美國決策者優先考慮與台灣的雙邊投資和貿易協定；CSIS在5月發表前美國駐港澳總領事唐偉康（Kurt Tong）的文章，其中主張美台達成貿易協定，將能加強美國在亞洲領導地位。

倡售台集束炸彈抗中

哈德遜研究所則於5月向美國決策者提出售台「集束炸彈」，以便台灣可以威脅對中共具有重大政治價值的非軍事目標及領導人，並建議鼓勵台灣有效威脅中國的非軍事目標。還鼓勵美國決策者「做出直接有利於台灣綠營執政黨的政策」，敦促美國決策者承認「一個自治或獨立的台灣」。

原文請見：

Taiwan funding of think tanks: Omnipresent and rarely disclosed

https://responsiblestatecraft.org/2020/06/17/taiwan-funding-of-think-tanks-omnipresent-and-rarely-disclosed/

[節錄]

Why not disclose?

Hudson may be the most extreme in its policy proposals, but the consistent behavior from the five think tanks is unmistakable: General support funding from Taiwan’s government is never disclosed when experts, whose salaries may well be partially funded by TECRO dollars, offer policy recommendations regarding U.S.-Taiwan relations.

“My philosophy is that if you’re producing any report, you should put right up front in an acknowledgment section that lists the specific funders, including general-support funders, that helped make this report possible and list any potential conflict of interest with the funders,” said Freeman of the Foreign Influence Transparency Initiative. “Let the reader judge for themself whether there’s a conflict of interest.”

Though the appearance or possibility of a conflict of interest does not mean that the funding flows from Taiwan influenced the work products produced by the think tanks, the decision not to prominently disclose the funding may undermine otherwise valuable analysis and policy proposals.

“It seems like because they hid it, they have something to hide,” said Freeman. “When the public trust in government is at all-time lows and people think D.C. is so corrupt, it’s even more important for think tanks and think tank scholars to put this information out there and try and restore the trust of the American public.”

Indeed, as acceptance of a cold war posture toward China becomes ever more accepted as a foregone conclusion by Washington influencers, one of them actually highlighted the danger of foreign funding going largely undisclosed.

CNAS’s 2020 report that advocated for a U.S.-Taiwan trade agreement warned of think tanks receiving “substantial funding from Beijing that is often targeted at shaping views and discourse on China.” CNAS recommended “higher degrees of transparency” to help “ensure that this funding is not generating hidden forms of foreign lobbying, self-censorship, or other activities that undermine core U.S. democratic principles.”

That self-awareness about the potential influence of foreign funding, and the ethical arguments for greater transparency, does not appear to extend to the omnipresent funding stream from Taipei to think tanks across the Beltway.

【After Winning Majority in LegCo: Beijing's Crackdown May Trigger International Intervention】

***感謝Hong Kong Columns - Translated，將我早前撰寫『議會過半想像：以「#國際攬炒」反制「臨立會2.0」』長文（https://www.facebook.com/joshuawongchifung/photos/a.313299448762570/2887650867994069/）翻譯成英文，鼓勵國際社會關注立會選舉一旦過半的沙盤推演，在最惡劣形勢下的制衡策略。***

中文精簡版本：https://www.facebook.com/joshuawongchifung/photos/a.564294826996363/2888641404561682/

Hongkongers have experienced our revolution for over half a year. They no longer take a consequentialist view to the effectiveness of their movement as they did years ago, or waste time second-guessing the intentions and background of fellow activists. Following the defensive battles at CUHK and PolyU, November’s District Council election saw a great victory of unity. More marvellous is the union between peaceful and “valiant” protesters.

In the process of resisting tyranny, the people have realised that one cannot prioritize one strategy over another. This is also how the common goal of “35+” came into being—the hope that we will win over half of the seats in the Legislative Council (LegCo) this September, such that the political spectrum that represents the majority of Hongkongers is able to gain control of legislative decisions. The political clout of Hongkongers will increase if 35 or more seats are successfully secured on our side. It is certainly one vital step to achieve the five demands within the system.

The possibility of realizing legislative majority

Technically it is not unrealistic to win a majority even under the current undemocratic system. Back in the 2016 LegCo election, we already won 30 seats. In addition to the District Council (First) functional constituency seat that is already in the pocket of the pan-democrats, as long as the candidates in Kowloon East and New Territories West do not start infighting again, we could safely secure 33 seats based on the number of pan-dem votes in 2016.

The other 3 seats required to achieve a majority depend on democrats’ breakthrough among the functional constituencies by dispersing the resources of the Liaison Office. They also count on whether the turnout this September could exceed 71.2% — that of last year’s District Council elections. Some of the factors that could affect the turnout include: will the epidemic persist into the summer? Will there be potential violent repression of protests in the 2 weeks preceding the election? Will Hong Kong-US relations be affected by the downturn of the global economy?

Therefore, the ambition of “35+” is to be prioritised by the resistance as both a means and an end. I have already expressed my support for an intra-party primary at the coordination meeting. In the meantime, it is pleasing to see the ongoing debates reaching a consensus of maximising the seats among geographical constituencies in the upcoming election.

Whilst enthusiastic coordination, we should also assess the post-election landscape and gauge Beijing’s reactions: if we do not reach 35 seats, Hong Kong will be subject to tighter control and more severe repression by China; but if the democratic parties successfully form a majority in LegCo, CCP’s fears of a “constitutional crisis” would become imminent. Hence, the key questions are how the Pan-Democrats should deal with the volatile political situation in Hong Kong and how they are going to meet Beijing’s charge head-on.

Watching out for Beijing’s dismissal of LegCo after reaching majority

To take back control of LegCo such that it faithfully reflects the majority’s principles and needs is the definition of a healthy democracy. Recently, however, DAB’s Tam Yiu-chung has warned that the plan of the Pan-Dems to “usurp power” in the LegCo would only lead to Beijing’s forceful disqualification of certain members or the interpretation of the Basic Law. This proves that winning a majority in LegCo is not only a popular conception but also a realistic challenge that would get on the nerves of Beijing. Could Beijing accept a President James To in LegCo? These unknown variables must be addressed upon achieving a majority.

While there is no telltale sign as to Beijing’s exact strategy, we are already familiar with the way CCP manipulated the Basic Law in the past 4 years. Having experienced three waves of disqualifications in LegCo, twice kicked out of LegCo with my team, and thrice locked up in jail, I have no false hopes of an easy compromise from Beijing: they would not let Pan-Dems control LegCo for half a year and wait (as is the proper procedure) until after having negatived the Budget to dissolve the legislature, and thereby giving them an easy victory in the re-elections. The greater the Pan-Dems threaten Beijing’s rule in Hong Kong, the more likely that it will trigger Beijing’s repression.

Since the disqualification and arrest of lawmakers have already become “normalised”, one can even imagine the police stepping into the LegCo building to force Pan-Dems into voting. Neither is it beyond our imagination to expect the CCP to kick out all 70 lawmakers in a fit of rage and replace them with a provisional LegCo “2.0” [HKCT note: The first was from 25 Jan 1997 to 30 Jun 1998]. To depend on a majority that could lead to a chapter of a “new testament” for One Country, Two Systems is perhaps what many elites long for, but they are overly optimistic:for a ticket to the promised land will not be available at the Chief Executive election campaign a year and a half later.

Admittedly, the Pan-Dems cannot unilaterally initiate “Laam-chaau” [HKCT note: mostly translated into “scorched-earth” mentality or “mutual destruction”; some even translated into “If I burn, you burn with us”]. The most they can do is to force a standstill of the government, and not for long the LegCo will have been eliminated from the equation to make the wheels turn again. It all leaves the plan of “Negativing the motion → Dissolving LegCo → Re-election after re-election → the stepping down of Carrie Lam” merely as overly positive speculation, probably resulting from their overestimate of CCP's capacity for rational calculation. The Pan-Dems must guard their frontlines and recognise what the biggest threat from Hong Kong to China could be. In this case, should LegCo sessions be disrupted or suspended, the Pan-Dems would have to be well prepared to surmount the expected obstacles and prevent the disqualification crisis 4 years ago—a Catch-22 indeed.

Productive tension from global intervention: Using Laam-chaau against the CCP

What aggravates the CCP the most is the potential threat to Hong Kong’s unique status as the one and only “separate customs territory”. Any miscalculation will compromise its role as the Chinese economy’s “white gloves”. Imagine if CCP were to disqualify all 70 elected lawmakers and convene a meeting north of the Shenzhen River to pass a resolution to Hong Kong’s affairs (much like the Provisional Legislative Council “1.0" in 1997), how great will the shock be in a world with an effective Hong Kong Human Rights and Democracy Act? However hard it is to predict the future one thing is certain: With the US presidential election just around the corner, blows to the separation of powers would not be tolerated, and the West would necessarily effect countermeasures against the Hong Kong government.

Beijing has been relying upon Hong Kong to navigate the international community for decades. While clamping down on the political freedom of the cosmopolitan city, Beijing desires to maintain the financial centre’s economic freedom. Hence, we started lobbying for the Hong Kong Human Rights and Democracy Act four years ago, and today we are determined to promote “Laam-chaau” on an international scale.

The will of the voters must be reflected in an election. If a “35+” legislature were to be dismissed and replaced, such flagrant violation of democracy would assuredly cause a greater backlash than the infamous extradition bill. Knowing what the reality ahead of us could be, we have to combine our election and international strategies to oppose the placement of a 35+ LegCo with an “Emergency Legislative Council 2.0”, to advance an international “Laam-chaau” to Hong Kong’s status as “separate customs territory”. Only then will we stand a chance to resist the regime and to realise the five demands.

Adjusting our mindset: Overcoming the “constitutional crisis” to reach a resolution

Upon the realization of the “35+” LegCo, it is expected that the CCP will launch a devastating counterattack. The Pan-Dems should not expect LegCo to run normally; neither can the lawmakers realise their governing blueprints they have for Hong Kong. Rather, candidates will be able to compete against one another with visions of a liberated Hong Kong through popular vote. Bringing this point up has nothing to do with undermining the common goal of reaching a majority in LegCo, but rather channels the battle of LegCo to positive use upon the rule of law’s death and a “constitutional crisis” ahead. Knowing that Hongkongers have nothing to fall back on, all Pan-Dems should not miss the only way to the realization of “35+”.

Thus, be they partisans, nonpartisans, incumbent politicians, amateur politicians, or the civil society as a whole – if we stay in the political discourse of 2016 and continue to perpetuate old stereotypes, that is to deal with the divisions on the pan-democratic camp by favouring the most “local” faction; to consider only resource allocation and self-aggrandizement as the purpose of a LegCo campaign; to ignore how potential lawmakers are fitted to what specific roles; to turn a blind eye to the journey of resistance since last summer (extending indefinitely into the future)—They would lead as astray and cost us lose a precious opportunity for change by winning a 35+ majority.

The extent to which the pan-democrats can stay united in light of the political atmosphere since last summer is another problem that our side must to address. Before the watershed moment of 12th June 2019, many democratic delegates were trapped in the mentality of needing to “preserve people’s livelihood”, “be content of what we have accomplished”, and other strategies that favours stability. As the government refuses to heed to the five demands, whether the democrats, especially those in the functional constituencies, have the political will to go all-in is the real difficult question that confronts us in the upcoming LegCo election.

All in all, if “35+” cannot be realised, it is unsurprising to see LegCo being more heavily suppressed in the next 4 years; even if "35+" is achieved, it is questionable whether the pan-democrats are able to weather multiple attacks, verbal or physical, from the regime (judging from its power in the last four years) and utilise the international Laam-chaau strategy against the displacement of LegCo. Adhering to the motto of “we fight on, each in his own way”, I can only hope that Hongkongers in elections, street confrontations and international front can reconcile with each other, so that we may collectively compel the government to yield to our demands in the next six months. It is only by reaching a resolution before a real constitutional crisis that we can combat the institutional violence of the regime and not be devoured by it.

https://hkcolumn.blogspot.com/2020/04/joshua-wong-after-winning-majority-in.html?fbclid=IwAR216gf53pG_j9JOpDfr2GItvjLfrFSekKTPzoEs3-s9KBqvPEwz865P8vw