全面画像，而非简单指标.pdf

1Web of Science | Profiles, not metricsJonathan Adams、Marie McVeigh、David Pendlebury及Martin Szomszor合著2019年1月全面画像，而非简单指标 。作者简介Jonathan Adams教授是科睿唯安旗下科学信息研究所（ Institute for Scientific Information ，简称ISI ）的负责人。他是伦敦国王学院政策研究所的客座教授。由于在高等教育和政策研究领域的卓越贡献， Jonathan Adams在2017年被埃克塞特大学授予荣誉理学博士学位。Marie McVeigh是科学信息研究所编辑团队的编辑伦理主管。她最初是宾夕法尼亚大学的细胞生物学家。自1994年以来， Marie McVeigh在科学信息研究所（ ISI）及其前身机构任职，致力于期刊管理和文献情报工作并发表了一系列相关文章。她近期负责的期刊引证报告（ JCR）的提升工作，增加了文章层面的具体信息和数据透明度，以支持正确使用期刊引证指标。关于科学信息研究所（ ISI）科学信息研究所是科睿唯安旗学术研究事业部（ Web of Science Group，简称WoSG ）的附属“研究院”，负责维护公司知识库，该知识库用于构建Web of Science及相关信息、分析内容、产品和服务； ISI通过活动、会议和出版物等形式对外进行知识传播，并开展研究以维持、扩展和改进知识库。David Pendlebury是科学信息研究所的研究分析主管。自1983年以来，他一直致力于使用Web of Science数据来探寻科学研究的结构和动态。他与ISI创始人Eugene Garfield共事多年，并与Henry Small共同开发了基本科学指标（ Essential Science Indicators）数据库。Martin Szomszor博士是科学信息研究所研究分析主管。他曾是数据科学（ Data Science）负责人，以及全球研究机构识别数据库（ Global Research Identifier Database）创始人，他将机器学习、数据集成和可视化技术领域的广泛知识应用到相关工作中。他因与英格兰高等教育资助委员会合作创建了“ REF2015具有影响力案例研究数据库”，荣膺“ 2015年英国信息时代50强数据领袖”称号。关于Web of ScienceWeb of Science是全球最值得信赖的、最大的、非出版机构的引文索引数据库平台，助力全球自然科学、社会科学及人文艺术领域的学术发现和引文分析。从政府部门到学术机构再到研究型企业 ， Web of Science每天为数百万用户提供可追溯自1900年的超过14亿条引文数据。 Web of Science为期刊影响因子（ Journal Impact Factor）、 InCites和其他强大的、可信赖的的引文影响力指标提供数据基础。Web of Science可帮助科研人员、研究机构、出版商和基金组织发现和评估来自权威期刊、书籍和会议录的，拥有百年以上历史的研究文献的引文影响力。想了解更多信息，敬请访问： clarivate/products/web-of-science科睿唯安 全面画像，而非简单指标 1本报告强调: 当有关科研人员及其机构的数据被压缩为简单的指标或排名时，信息将会丢失。本报告阐述了四类常见分析，如若误用将掩盖真实的科研表现；我们提出了四种可视化选项，用于解读每个度量指标下蕴含的更丰富的信息，以及支持开展全面的、负责任的科研管理。我们身边依然存在着声称可通过简单分析来评估论文、科研人员和机构表现的现象。尽管资深分析专家提出反对意见，诸多研究人员对此深表忧虑，但是大学管理人员依然乐此不疲。有关大学排名可信度的争议无休无止，但大学排名并不会因此而停止发布。我们不禁要问：诸如单点指标和线性排名之类的简单分析为何如此受到欢迎？总结性统计数据和排名表拥有与生俱来的魅力。我们希望“看看谁做得最好”，就像观看体育比赛一样。但是，联赛积分表是既定成员之间通过进行一系列比赛而衍生的产物，谁在公开比赛中的综合成绩更好，谁的近期排名就越靠前。联赛积分表基于一维性的比赛名次，能够一目了然地显示出选手通过博弈而取得的排名。但科学研究并非一维性事物：其过程很复杂，没有两个项目是完全一致的。研究机构也并非只有一项任务：他们既要搞教学，又要搞研究；他们的研究内容可能是基础性、分析性、应用性、协作性、社会性或行业性的；他们的研究活动可能跨越多个学科，每个学科都有各自的学术特点。单点指标在某些类型的比较中是具有价值的，例如各大学类似的院系中每名研究人员的相对产出，我们从中可以了解到 “类似”研究中的真正差异。但是，如果用单点指标替代全面的科研管理，例如在缺乏补充信息的情况下进行学术评估，甚至将单点指标视为招聘标准，那么，这类信息就具有一定的局限性，而且单个（或孤立）指标可能会被误用。通过使用分布在科研活动和学科领域中的替代性指标，大学排名采用一系列变量来“描绘”一所大学。每个变量都被标引：与计数、资金、影响力、时间及其他不兼容的项目相挂钩而进行量化；然后再通过加权将不同的项目组合在一起，从而得出最终分数。如果没有合理的数据管理机制，该分数将远远不能体现大学生活的丰富性与多样性。每一个被过度简化或误用的指标，其实都有更好的替代选项。一般是先进行适当的、负责任的数据分析，再以图形方式来显示多个互补的维度。通过展开数据，将指标置于某个背景下，或将其置于更加广泛的场景中，我们能够看到新的特征，并能了解更多信息。下面的示例将充分显示这个举措是多么简单易行，而且能大幅提高我们解读科研活动的能力。Web of Science | Profiles, not metrics 1We are surrounded by analyses that claim to measure relative performance among people and organizations. University managers evidently use them, disregarding counter-arguments offered by informed analysts and to the dismay of researchers. Critical discussion about the credibility of university rankings is endless, but they continue to be published. We ask: nullhy are simplistic analyses, such as single-point metrics and linear rankings, so popularnullnullummary statistics and league tables have innate appeal. We nullant nullto see nullho does bestnull, taking an analogy from sports. nullut a sportsnullleague table is the product of a series of matches betnulleen similar members of a defined group, nullith the table headed up by nullhichever currently has the better balance of victories in direct and enullplicitly matched competition. nullleague table is a one-dimensional ranking based, sensibly for its specific purpose, on the single dimensions of the paired matches.nullesearch is not one-dimensional: the process is complenull and no tnullo pronullcts are identical. nullor do research organizations have a single mission: they teach as nullell as researchnulltheir research may be blue-skies, analytical, applied, collaborative, societal or industrialnulland their activity is spread across many disciplines, each nullith its onulln academic characteristics.nullngle-point metrics have value nullhen applied in properly matched comparisons, such as the relative output per researcher of similar research units in universities. nullhat may tell us about real differences in nullsimilarnullresearch. nullut the information is limited and an individual nullor isolatednullmetric can be misused if it is a substitute for responsible research management, for enullample in academic evaluation nullithout complementary information, or even as a recruitment criterion.University rankings take a set of variables to nullpicturenullan organization, using pronully data spread across activities and disciplines. nullach variable is indenulld: scaled to link counts, money, impact, time and other incompatible itemsnulland then nulleighted to bring different items together in a final score. Without nullell-informed data management, that number may have only a distant relationship to the rich diversity of university life.nullor every over-simplified or misused metric there is a better alternative, usually involving proper and responsible data analysis through a graphical display nullith multiple, complementary dimensions. nully unpacking the data and placing the metric against a background or setting it in a nullider contenullt, nulle see nenullfeatures and understand more. nullhe enullamples that follonullshonullhonull easy this is and honullmuch it improves our ability to interpret research activity.nulln this report, nulle dranullattention to the information that is lost nullhen data about researchers and their institutions are snullueezed into a simplified metric or league table. We look at four familiar types of analysis that can obscure real research performance nullhen misused and nulle describe four alternative visualizations that unpack the richer information that lies beneath each headline indicator and that support sound, responsible research management. 科睿唯安 全面画像，而非简单指标2个体： h指数与射束图物理学家Jorge Hirsch于2005年创建了h指数， h指数是一种被广泛应用但却不一定能够全面反映科研人员论文和引文影响力的指标。它将一系列论文及其被引次数缩减为单个数字：一个具有指数h的研究人员（或团队，甚至是国家）至少已发表了h篇论文，并且每篇论文至少已被引用了h次。h指数取决于职业生涯的长度和学科。因为随着时间的推移，不同研究领域之间的论文，其被引次数的积累速度各不相同。因此， h指数不适用于对个体进行比较；它通常不涵盖非期刊出版物；从数学的角度看，也不具有一致性（ Waltman和van Eck， 2012）。来自德国马普学会的Lutz Bornmann和Robin Haunschild提出的另一种方法（ Bornmann和Haunschild， 2018）：是将研究人员的文章放在适合进行比较的背景中。每篇论文的被引次数均按与其具有相同学科和出版年份的期刊的平均值进行“规范化”，并将该值转换为百分位数。相对单纯的平均值而言，这种方法因为引用分布偏斜度很大，从而能更准确地衡量集中趋势。百分位数为90意味着该论文位于引用率最高的前10%之列，另外90%则是引文影响力较低的论文。中位数为50：论文的篇均影响力范围基本在0到100之间。射束图可用于开展公平的、有意义的评估。它可迅速传达h指数永远不能传达的信息。如图所示，该位研究人员的平均百分位数明显高于50 ，表明其发表的论文在其所属领域具有核心影响力。虽然论文的年度中位影响力在早年低于50 ，但我们可以看出，这一数值随着时间的推移逐渐超过了平均水平。Web of Science | Profiles, not metrics2Individuals: h-index vs the beam-plotA widely quoted but poorly understood way of characterising a researchers publication and citation profile is the h-index, created by physicist Jorge nullirsch null2nullnullnullnullnullIt reduces a list of publications and their citation counts to a single number: a researcher nullor group or even countrynullwith an index of h has published at least h papers each of which has subsequently been cited at least h timesnullnullhe h-index depends on career length and discipline because citation counts accumulate over time at rates that vary between research fields, so it provides no proper comparability between individualsnullit usually excludes non-nullurnal publicationsnulland it is mathematically inconsistent nullnullaltman and van nullcnull, 2nullnull2nullnullAn alternative approach proposed by nullutnullnullornmann and nullobin nullaunschild, nullax nulllancnullInstitute nullnullornmann and nullaunschild, 2nullnullnullnull, puts a researchers articles into a context suitable for comparisonnullnullach papers citation count is nullnormalinulled by the average for nullurnals in their same category and publication year, and that value is converted to a percentilenullnullhis provides a better measure of central tendency than an average because citation distributions are so snullewednullA percentile of nullnullmeans that a paper is among the nullnullnullmost cited and the other nullnullnullhave achieved less citation impactnullnullhe median score is nullnull: the average impact among publications ranged within a common scale between nulland nullnullnullnullnullhe beam-plot can be used for a fair and meaningful evaluationnullIt quicnullly conveys information that the h-index never suggestednullnullhis researchers average percentile is signinullcantly better than nullnull, the central impact in the nullelds where they publishednullnullheir median annual impact was below that benchmarnullin early years but can be seen to move above the average over timenullFigure 1nullIn this example h-index null2nullfor a researcher who is an author or co-author on nullnullcitable nullournal articles over a nullnull-year periodnullnullutput included reports and proceedings that cannot be analysed in this waynullnullraphing the data reveals the spread, snullw, and presence of relatively highly cited items buried under the nullh valuenullnullncited items disappearnullFigure 2nullA beam-plot of the data in nulligure nullnullnullach article is compared to its own reference set but all use a common null-nullnullnullpercentile scalenullnullhe ranges of each years article percentiles are shown nullgrey marnulls, across the beamnullwith their annual median nullpurple marnull, a pivotnullnullnullhe benchmarnullline is the researchers overall average: the nullnullthpercentilenull02550751001 11 21 31 41Citationsperpaperat2018Papers ordered by citation count23The h-index of the papers in this graph is 23That is: 23 of 44 papers by this researcher have been cited 23 or more times since publication23Web of Science | Profiles, not metrics2Individuals: h-index vs the beam-plotA widely quoted but poorly understood way of characterising a researchers publication and citation profile is the h-index, created by physicist Jorge nullirsch null2nullnullnullnullnullIt reduces a list of publications and their citation counts to a single number: a researcher nullor group or even countrynullwith an index of h has published at least h papers each of which has subsequently been cited at least h timesnullnullhe h-index depends on career length and discipline because citation counts accumulate over time at rates that vary between research fields, so it provides no proper comparability between individualsnullit usually excludes non-nullurnal publicationsnulland it is mathematically inconsistent nullnullaltman and van nullcnull, 2nullnull2nullnullAn alternative approach proposed by nullutnullnullornmann and nullobin nullaunschild, nullax nulllancnullInstitute nullnullornmann and nullaunschild, 2nullnullnullnull, puts a researchers articles into a context suitable for comparisonnullnullach papers citation count is nullnormalinulled by the average for nullurnals in their same category and publication year, and that value is converted to a percentilenullnullhis provides a better measure of central tendency than an average b cause citation distributions are so snullewednullA percentile of nullnullmeans that a paper is among the nullnullnullmost cited and the other nullnullnullhave achieved less citation impactnullnullhe median score is nullnull: the average impact among publications ranged within a common scale between nulland nullnullnullnullnullhe beam-plot can be used for a fair and meaningful evaluationnullIt quicnullly conveys information that the h-index never suggestednullnullhis researchers average percentile is signinullcantly better than nullnull, the central impact in the nullelds where they publishednullnullheir median annual impact was below that benchmarnullin early years but can be seen to move above the average over timenullFigure 1nullIn this example h-index null2nullfor a researcher who is an author or co-author on nullnullcitable nullournal articles over a nullnull-year periodnullnullutput included reports and proceedings that cannot be analysed in this waynullnullraphing the data reveals the spread, snull d presence of relatively highly cited items buried u er the nullh valuenullnullnci d items disappearnullFigure 2nullA beam-plot of the data in nulligure nullnullnullach article is compared to its own reference set but all use a common null-nullnullnullpercentile scalenullnullhe ranges of each years article percentiles are shown nullgrey marnulls, across the beamnullwith their annual median nullpurple marnull, a pivotnullnullnullhe benchmarnullline is the researchers overall average: the nullnullthpercentilenull02550751001 11 21 31 41Citationsperpaperat2018Papers ordered by citation count23The h-index of the papers in this graph is 23That is: 23 of 44 papers by this researcher have been cited 23 or more times since publication23按被引次数进行排序的论文百分位数（ 0最差， 100最好）图1.在此示例中， h指数 = 23的科研人员在15年间共发表了44篇可被引用的期刊文献（单独编著或与其他人合著），其中包括不能以这种方式进行分析的报告和会议论文。通过对数据制图，我们可以发现隐藏在“ h”值下的相对高被引文章的分布、偏斜和存在情况，及消失的未被引用的文献。图2.图1中数据的射束图，将每篇文章与相应年度与学科的参考数据集进行比较，都使用0-100百分位数。该图显示了每年的百分位数范围（射束灰色标记）及其年度中位数（枢轴上的紫色标记）。基准线是该研究人员的总体平均值：百分位59 。图中论文的h 指数是23表明：该研究人员44篇论文中的23篇，自发表后至少被引用了23次出版年份每篇文章在2018年的被引次数科睿唯安 全面画像，而非简单指标 3图3.左图： EMBO Reports的期刊影响因子趋势图显示了该期刊的影响因子变化趋势及其在相应学科全部期刊中的影响因子排序百分位变化趋势。右图： 2017年引文分布图显示了中位数和整体分布情况。期刊：期刊影响因子（ JIF）与期刊引证报告（ JCR）的对比定量科研评估通常会关注出版物的集合，并将平均被引次数与基于学科的基准进行比较。评估人员还可以关注发表该文章的期刊。期刊影响因子（ JIF）是我们所熟悉的常用指标。 JIF由科学信息研究所（ ISI）创始人Eugene Garfield创建。 Garfield于1955年提出了出版物“影响力”的概念并创建了“期刊”影响因子（ Garfield and Sher， 1962），旨在帮助全新的科学引文索引（ SCI）选择新期刊。第一份期刊引证报告随后于1975年正式出版。JIF2（即基于两年的期刊数据）有两个基本要素：分子和分母。分子是指前两年在期刊上所发表的任何论文在当前年份中的被引次数；分母是指这两年所发表的实质性的科研论文（ article）及综述（ review）的数量。这两个基本要素可通过调整来适应更短或更长的时间间隔。仅以前一年论文情况为考量的JIF可凸显快速变化的领域；而以前面5年或10年的论文情况（数量及被引次数）为考量的JIF则更能彰显某个特定年份的期刊被引用情况。预算有限且需管理诸多图书订阅工作的图书馆员通常使用定量期刊比较工具，出版商则使用该工具来跟踪系列出版物的绩效表现。问题在于，旨在科学管理期刊的JIF ，却被不负责任地应用于更广泛的研究管理之中。为解决这一问题， 2018年的期刊引证报告通过更为丰富的数据背景修订了期刊画像。例如：以同一学科中百分位数来表达JIF值的条形图将能够快速显示其分区；此外，对研究人员至关重要的是，每篇文献对引文的贡献都将在覆盖所有文章的引用频率分布图中进行显示。新的期刊画像清晰地表明， JIF将更大、更复杂的数据池汇总在一起。 JIF对期刊经理而言可能是个很有用的工具。但对研究管理人员而言， JIF只能提供他们需要的关于期刊或论文价值的部分信息。Web of Science | Profiles, not metrics 3Journals: JIF vs JCR distributionsQuantitative research evaluation usually looks at collections of publications and compares average citation counts with category-based benchmarksnull nullvaluators may also look at the nullurnals in which the articles are publishednullnullhe Journal Impact Factor (JIF) is a familiar indicator in general usenullIt was developed by nullugene nullrfieldnull the founder of the Institute for Scientific Information nullrfield nullnullnullnullnullnullraised the idea of publication nullimpactnull and created a nullnullurnalnullimpact factor nullnullrfield and nullhernull nullnullnullnullnullto help select nullurnals for the new Science Citation Index (SCI)nullnullhat anticipated the first Journal Citation Reports in nullnullnullnullnullJIF2 nullthat isnullbased on two years of nullurnal datanullhas two elements: the numeratornullthe number of cites in the current year to any items published in the nullurnal in the previous two yearsnulland the denominatornullthe number of substantive articles and reviews published in tho