Marina Cortês
Ticks of time: on Cosmology, Everest, and Ballet
Ticks of time: on Cosmology, Everest, and Ballet
I am part of a discussion collective of intellectuals brought together by Silicon Valley tech companies, jointly with companies in academic and scientific publishing. The discussion group is simply a town hall where members from very distinct academic and technological backgrounds can bring up and discuss any developments in technology, science or industry.
This might include the discussion of key ongoing societal, or economical, transformations anywhere on the planet. Through this discussion group I was aware of the chatGPT technology mainly since something like the Summer of 2022. However I did not pay proper attention until December 2022.
Roughly by mid April 2023, I had learnt enough about the ongoing events, to become convinced that such developments merited devoted attention by academics. I decided to use my own expertise as a physicist, and data analyst, to look into the generative AI technology, its software, and the data being acquired, in order to be able to assess for myself the capability and behaviour of the upcoming technologies.
Here is a step by step itemisation of the developments ordered by date. They contain the sources I found the most useful, as 2023 unfolded and development of the generative (large language) models were rolled out.
January 20th, 2023 — Google Layoffs include many voices of Open Source software, including Chris DiBona (Director and Founder of Google’s Open Source Program Office, OSPS), Cat Allman (Google Open Source Program manager), Mike Frumkin (Founder of Google’s Accelerated Science Team).
The layoffs happened suddenly, announced by email, without any notice, and included some of the strongest, most brilliant individuals in the company. I received this news very seriously as did many of us in these circles. Most unsettling was the realisation that some of the most talented minds, DiBona, Frumkin, Allman—that have worked inside big tech towards just and open technological societies for decades—could lose their positions as gatekeepers of software security and trustworthiness in the very groups they had founded.
Personally this was the first alert that a new, significant, and disquieting change inside big tech, might be about to happen, and most likely already underway. Knowing personally the quality of the individuals in question gave me a good grasp on the magnitude of power that was being called upon in order to carry through these layoffs and implement and bring forth the new technologies.
January 20th, 2023 marks for me a turning point in realising that a change in trajectory of the planet was taking place. I remain hopeful that we will be able to reverse this trajectory, as societies and as a whole.
“The AI Dilemma“: Center for Humane Technology, March 9th, 2023, with Tristan Harris and Aza Raskin.
This video contains the most up-to-date source of information (until August 6th, 2023–scroll below for more info). This is also in my view the most authoritative and well prepared content. Harris and Raskin have on March 9th, already anticipated many of the developments that have taken place subsequently– at least 5 months ahead of their time.
Key moments:
1. 2023 is the year when all content-based identification will break down.
2. The technology is in accelerated development. We do not currently have a means for measuring the rate of acceleration.
3. Extractive technologies.
Here are the citations mentioned in The AI Dilemma:
Emerging Tech Trend Report, 2023, Amy Webb at SXSW 2023, March 29th, 2023.
“Futurist Amy Webb, CEO of the Future Today Institute and professor at NYU Stern School of Business, provides a data-driven analysis for emerging tech trends, and shows perspective-changing scenarios for the future.”
“There is no AI“, Jaron Lanier (founding father of Virtual Reality) April 20th, 2023, New Yorker Article.
“Risk-based methodology for deriving scenarios for testing artificial intelligence systems” by Barnaby Simkin – NVIDIA – April 2023
Best derivation of principles for AI regulation I have seen till today (September, 5th, 2023), as far as a concrete strategy and clear outline are concerned.
My own opinion is that the interests of industry’s stakeholders need to be taken into account as well as the civil society’s interests and policy makers’ interests.
I believe we need to bear everyone’s interests in mind, during regulation, otherwise cooperation will not succeed. We must not overlook the financial interests of those we are trying to regulate or they will side step and assign the whole regulatory enterprise to legal limbo.
“How humanity can defeat AI“, Jaron Lanier, interview by UnHerd channel, May 5th, 2023.
Key moments:
1. Lanier states that despite being employed by Microsoft he has an agreement with the company in which he is free to speak his mind, but also does not speak for Microsoft. He enjoys academic freedom, with regards to the technologies under discussion.
2. Jaron states that the mathematics behind the large language models is “embarrassingly simple”. This is essentially the product rule of likelihoods (used in basic statistics) and is confirmed by Perimeter Institute’s Roger Melko’s lecture in May 2023, posted below.
The complex behaviour of the language models is a sign of the large number of free parameters to be fitted—the long file of order 10^{12} weights, obtained when the models are trained— as well as some clever ways to interconnect those degrees of freedom.
Perimeter Institute for Theoretical Physics‘ Roger Melko – computer science – May 2023.
Melko, R. (2023). LECTURE: Generative Modelling.
3 days of lectures of technical derivation of the mathematical machinery behind large language modelling (generative AI). Good introduction for physicists and data analysts.
May 8th, 2023.
DOI : 10.48660/23050140
40mins – in Large Language models the bottle neck is the training cost. GPT3 cost USD 20 million to train. GPT4 (not disclosed) possibly USD 100 million cost to train.
42mins – Roger agrees that the number of parameters in learning technologies must not exceed the quantum gravitational cap on information stated by the Black Hole entropy result.
May 9th, 2023.
DOI : 10.48660/23050097
May 10th, 2023.
DOI 10.48660/23050095
Min 38: Melko starts to explains the architecture mathematics behind LLMs
Min 39: Gives the mathematical rule for the joint distribution estimator of the data vector, v, (visible units). The estimator maps the visible units of the data vector to a sequence, using the chain rule of probabilities. This portrays the autoregressive property of the models, and is one of the most powerful properties of LLMS.
Min 40: Sec 31 — Using a Stars Wars example, “ May the force be with you” Melko provides the simplest explanation for the behaviour of the functioning of the LLM I have seen to today. LLMs are probabilistic reasoning.
Summary:
Melko describes how the technology involved in large language pre-trained models is a predictive text technology, which operates on a word-by-word basis. In particular Melko explains that LLMs are overparametrized / under-fitted. This means that the number of free-parameters in a fitting model is larger than the number of parameters that the available data requires to be fit to, or explained. In a typical pre-trained model there is not enough (natural/digital) data to estimate the likelihood function of the data distribution by usual MCMC methodology. As a result the statistical rule used to calculate the likelihood of a given data vector is the chain rule of products.
White House “AI risk management Framework”, National Institute of Standards and Technology (NIST) by US, Department of Commerce. White House release of the 2023 updated National AI R&D Strategic Plan.
AI risk, a view from ex-googler Mo Gawdat, June 1st, 2023.
Business Insider article.
Randy Fernando, Center for Humane Technology, AI Town Hall, June 7th, 2023. (An update to the earlier Harris-Raskin presentation above).
Meta VoiceBox `too risky to release’, June 16th, 2023:
press release,
research post,
academic research article (Facebook Research).
DisrupTV-327 discussion panel, June 24th, 2023
Panelists:
David Bray (Distinguished Fellow – Stimson Center and Business Executives for National Security)
Divya Chander (Anesthesiologist, Neuroscientist, and Data Scientist)
Megan Palmer (Senior Director for Public Impact at Ginkgo Bioworks and Adjunct Professor of Bioengineering at Stanford University).
AI development on filmmaking and acting, June 29th, 2023, The capabilities of the technology seem to have reached, or are about to reach seamless human-actor, voice, imagery, and footage, together with the insertion onto contemporary feature-length movies, as well as past ones.
“What my musical instruments have taught me”, Jaron Lanier, New Yorker, July 22nd, 2023. In this article Lanier states that reality is incompressible. In my view this implies that AGI is not likely to be achieved.
“If you work with virtual reality, you end up wondering what reality is in the first place. Over the years, I’ve toyed with one possible definition of reality: it’s the thing that can’t be perfectly simulated, because it can’t be measured to completion. Digital information can be perfectly measured, because that is its very definition. This makes it unreal. But reality is irrepressible.”
I wrote this post on Jaron’s article (August 2023) related to art and AI around the same time. Here is an earlier art and AI point of view from April 2023.
At the beginning of August 2023 a new turn of events would change the scenario of the extent of reach of transformative AI technologies. Namely online meeting platforms changed their terms and conditions to allow for freer inclusion of generative AI software in collaborative meetings.
Around August 6th, 2023 the online-meeting platform “Zoom Video Communications, Inc.” updated their terms and conditions. Amongst other new Zoom features this update enabled the inclusion and widespread dissemination of a collaborative technology known as “Otter AI“. Otter AI is a note-taking piece of software for usage in online collaboration (based in Mountainview, CA). The implications of this particular technology, particularly in what regards, its design, and default settings, have consequences, that as of now, I do not see we could have anticipated with the content that Harris and Raskin shared.
On August 7th, 2023, the platform representatives denied that this update of terms allowed for third-party model training on data content owned by the Zoom application.
Some background context regarding Otter AI and its history of interaction with the Zoom corporation (quoting from the platform’s website):
“The following changes have gone into effect on September 27th, 2022 for the Otter Basic plan.
OtterPilot will be included in the Otter Basic plan. Users will be able to have their OtterPilot automatically join meetings for Zoom, Microsoft Teams, and Google Meet to automatically record and transcribe in real-time. Users can easily access their notes, even if they can’t join the meeting. Learn more about OtterPilot.“
August 27th, 2023 — New-Text to-Video developments by One Prompt
Do pay close attention to the pace of development in the technology. This capacity was already hinted at by Emad Mostaque, of Stable Diffusion a few months ago in late March 2023, I paste the link of the discussion with Emad Mostaque at the conference Abundance 360 here in the next section, since I only saw it recently, in mid August.
Emad Mostaque at Abundance 360 March 20th-23rd, 2023 at the Abundance360 (A360) conference.
Emad Mostaque appears to be quite spot on and very sharp with content.
He gave this interview at the end of March, I highlight a few statement in bullet points here below, I did not get through the entire talk.
I was very surprised by his views held at the end of March this year, which are far ahead anything I was thinking at that time.
And a few bonus extras:
