Room: EDUC 114

Instructor: Vladik Kreinovich, email vladik at utep.edu, office CCSB 3.0404,
office phone (915) 747-6951.

• The instructor's office hours are Tuesdays and Thursdays 1:30-2:50 pm, or by appointment.
• If you want to contact the instructor during the scheduled office hours, there is no need to schedule an appointment.
• If you cannot contact the instructor during the instructor's scheduled office hours, please schedule an appointment in the following way:
  • use the instructor's appointments page https://www.cs.utep.edu/vladik/appointments.html to find the time when the instructor is not busy (i.e., when he has no other appointments), and
  • send him an email, to vladik at utep.edu, indicating the day and time that you would like to meet.
  He will then send a reply email, usually confirming that he is available at this time, and he will place the meeting with you on his schedule.

Main Objective: to teach methods of dealing with uncertainty in AI.

Contents

1) What Is AI. In order to talk about uncertainty in AI, we first need to discuss what is AI and what is not AI.

From the computational viewpoint, an ideal computational problem is when:

• we have a well-defined problem, and
• to solve this problem, we apply a well-justified algorithm to objective (measurement-based) data.

In many real-life situations, however:

• the problem is not well-defined (e.g., how to formalize "comfortable"),
• some data comes not from measurements but from expert estimates, and
• to process this data, we use expert-proposed heuristic (= not justified) methods.

So, in a nutshell, AI is using heuristics, using expert knowledge.

2) Why uncertainty? Uncertainty is everywhere in AI: heuristics are not precise, expert knowledge is not precise, measurements are not precise. So, we need:

• to quantify this uncertainty -- i.e., describe it in computer-understandable numerical terms, and
• to propagate this uncertainty through the algorithms, i.e., describe, based on the uncertainty with which we know inputs, what will be the uncertainty of the algorithm's output.

• taking into account additional uncertainty caused by the computational device: from round-off errors in the traditional computers to the probabilistic character of quantum computers, and
• taking into account uncertainty that we introduce ourselves on purpose, e.g., to preserve privacy.

3) Why not just use machine learning (ML). Current ML tools are so good that it seems like they can compute everything. Unfortunately, uncertainty is an exception: current ML tools do perform most computations very well, but they are not very good in estimating uncertainty -- in particular, uncertainty of their own results. For example, when they mistake a dog for a cat, they claim it with 99.9% confidence. There is a reason for this:

• when we train a system to predict the result, we can (and do) use thousands of examples, but
• when we train it to get the probability of each result, we need about 100 examples of computations to get one probability estimates -- so the number of inputs for such training is 100 times smaller, not enough to get good results.

4) Types of uncertainty.

• In the ideal case, we know all possible values of the corresponding quantity and we know the frequency with which each value occurs -- i.e., we have a probabilistic uncertainty.
• In other cases, we have no information about the frequencies -- this corresponds to interval or, more generally, set-valued uncertainty.

• cases when we know intervals of possible values of probability; this is known as probability boxes or, for short, p-boxes, and
• cases when we know the frequency of different possible probability distributions; this is known as the Bayesian approach.

• we can try to elicit subjective probability (also known as prior distribution) from the experts, or
• we can simply ask the experts to gauge their own degree of certainty, e.g., on a scale from 0 to 1; this is known as the fuzzy approach.

Instead of numbers, it often makes sense to elicit intervals -- or even fuzzy statements; this modification is known as type-2 fuzzy techniques.

5) Measures of uncertainty. There are two basic ways to measure uncertainty:

• We can view it from the viewpoint of amount of knowledge -- i.e., describe the amount of effort needed to complete our uncertain knowledge. This approach leads to entropy.
• We can also explicitly take into account the effect of uncertainty on our decisions, i.e., gauge losses caused by uncertainty. For example, if a company does not want to reveal its current sales amount to its competitors, disclosing the least significant binary digit (bit) of this amount and disclosing the first bit both provide the competitors with exactly 1 bit of information. However, the first disclosure is harmless, while the second can lead to serious losses.

6) How to come up with algorithms. Here, we will talk about standard data processing techniques, with an emphasis on the probability-based ones: Maximum Likelihood, its most used particular case -- Least Squares, and Markov Chain techniques. We will also talk about simulation, interpolation and extrapolation, robustness, and regularization.

7) How to propagate uncertainty, in particular through Large Language Models (LLMs) and simulation algorithms. This will be one of the main topics of this course, we will learn uncertainty propagation algorithms covering all types of uncertainty: probabilistic, interval, and fuzzy. Most of these algorithms are designed for propagating uncertainty through open-source algorithms, but we will also cover techniques for propagating uncertainty through proprietary black-box algorithms.

8) How to combine uncertainty. Often, different inputs come with uncertainty of different type: for example, we know the probability distribution of one of the quantities, and we only have expert (fuzzy) information about another quantity. For such cases, we will study two types of methods.

When most of the information is of the same type, we can simply transform the remaining different-type information to this type:

• we can use Maximum Entropy approach (MaxEnt) to transform intervals into probabilities,
• we can use defuzzification to transform fuzzy data into numbers,
• we can use midpoint (or Hurwicz approach) to transform an interval into a number, etc.

In situations when there is no dominant type of uncertainty -- or when we want to extract as much information from the uncertain data -- we need to use more sophisticated methods for actually combining different types of uncertainty.

9) Reasoning under uncertainty, causality, etc. In this section, we will be studying basic fuzzy techniques, basic statistical testing, and basic ideas of non-monotonic logics.

10) Decision making under uncertainty.

• We start with decision making in situations without uncertainty -- i.e., in mathematical terms, with optimization. We will study gradient descent -- the simplest numerical optimization technique that works so well in neural networls.
• Then we will study decision making under the most detailed type of uncertainty -- probabilistic uncertainty. We will study the utility approach -- the standard approach of decision theory.
• Then, we will deal with the case of interval uncertainty, where decision theory leads to the Hurwicz criterion.
• After that, we will study fuzzy decision making.
• Finally, we will study symmetry approach -- a physics-motivated approach which is useful when we have very little information. We will show how this approach explains many successful heuristics in fuzzy and neural networks.

11) Other topics. At the end of the class, we will briefly mentioned other important uncertainty-related topics, such as:

• uncertainty visualization (including relation between 256 distinguishable intensities and the use of 8-bit numbers on neural computing) and
• dealing with conflict situations under uncertainty.

Sources: On this topic, there is no up-to-date textbook yet, we will use several papers. For topic for which there are no easy-to-read papers, we will try to post easier-to-summaries of the not-so-easy-to-read papers on the class website.

Projects: An important part of the class is a project. There are three possible types of projects:

• For students who already have a research advisor, an ideal class project is if you do something related to uncertainty of AI which is useful for your research project and/or for your future thesis or dissertation. Please check with your advisor about it, maybe he or she wants you to read and report on some related paper, maybe you need to do some related research, whatever your advisor recommends will be a very good project for this class, just let the class instructor know what exactly you plan to do.
• If you do not have an advisor, find an interesting paper that is related to uncertainty in AI and that you would like to review, and send it to the instructor -- so that he will check that this paper is relevant and reviewing it is a doable task.
• You can also select one of the projects proposed by the instructor.

Assignments: Reading and homework assignments will be announced on the class website. You should expect to spend at least 10 hours/week outside of class on reading and homework.

Homework Assignments: Each topic means home assignments. Howeworks will be due by the day of the next class. They will be usually assigned on Tuesday, and due in 2 days, on Thursday. To submit a homework, send it to me by email. If it is not electronic, scan it and send him/her the scanned version.

One week after the homework was assigned, I will post correct solutions. I will be glad to answer questions if needed.

If you have a legitimate reason to be late, let me know, you can then submit it until the homeworks are posted. If you were simply late, you can still submit until the homeworks are posted, but then points will be taken off points for submitting late.

Since I will be posting correct solutions to homeworks, it does not make any sense to accept very late assignments: once an assignment is posted, it make no sense for you to copy it in your own handwriting, this does not indicate any understanding. So, please try to submit your assignments on time.

Things happen. If there is an emergency situation and you cannot submit it on time, let me know, you will then not be penalized -- and I will come up with a similar but different assignment that you can submit to me when you become available again.

Homework must be done individually. While you may discuss the problem in general terms with other people, your answers and your code should be written and tested by you alone. If you need help, consult the instructor.

Exams: There will be two tests and the final exam on December 9, 1-3:45 pm.

Similar to homeworks, I will post solution, send you the grades, and answer questions if something is not clear.

As usual, if you are unable to attend the test, let me know, I will organize a different version of the test at a time convenient for you.

Grades: Each topic means home assignments (mainly on the sheets of paper, but some on the real computer). Maximum number of points:

• first test: 15
• second test: 20
• home assignments: 10
• final exam: 35
• project: 20

A good project can help but it cannot completely cover possible deficiencies of knowledge as shown on the test and on the homeworks. In general, up to 80 points come from tests and home assignments. So:

• to get an A, you must gain, on all the tests and home assignments, at least 90% of the possible amount of points (i.e., at least 72), and also at least 90 points overall;
• to get a B, you must gain, on all the tests and home assignments, at least 80% of the possible amount of points (i.e., at least 64), and also at least 80 points overall;
• to get a C, you must gain, on all the tests and home assignments, at least 70% of the possible amount of points (i.e., at least 56), and also at least 70 points overall.

Special Accommodations: If you have a disability and need classroom accommodations, please contact the Center for Accommodations and Support Services (CASS) at 747-5148 or by email to cass@utep.edu, or visit their office located in UTEP Union East, Room 106. For additional information, please visit the CASS website at http://www.sa.utep.edu/cass. CASS's staff are the only individuals who can validate and if need be, authorize accommodations for students.

Scholastic Dishonesty: Any student who commits an act of scholastic dishonesty is subject to discipline. Scholastic dishonesty includes, but not limited to cheating, plagiarism, collusion, submission for credit of any work or materials that are attributable to another person.

Cheating is:

• copying from the test paper of another student;
• communicating with another student during a test to be taken individually;
• giving or seeking aid from another student during a test to be taken individually;
• possession and/or use of unauthorized materials during tests (i.e. crib notes, class notes, books, etc.);
• substituting for another person to take a test;
• falsifying research data, reports, academic work offered for credit.

• using someone's work in your assignments without the proper citations;
• submitting the same paper or assignment from a different course, without direct permission of instructors.

Collusion is unauthorized collaboration with another person in preparing academic assignments.

Instructors are required to -- and will -- report academic dishonesty and any other violation of the Standards of Conduct to the Dean of Students.

NOTE: When in doubt on any of the above, please contact your instructor to check if you are following authorized procedure.

Daily schedule (tentative and subject to change)

August 26: general introduction, see parts 1-4 of this syllabus; see also lecture on subjective probability

August 28: how to describe uncertainty - an approach based on decision making; see Slides 27-37 from slides ssci18, Section 5 of paper, tr 19-78, Section 3 of the paper tr07-53b, and

September 2: how to describe uncertainty - an approach based on decision making (continued)

September 4: measures of uncertainty; see Section 2 of paper 91-11

September 9: Maximum Likelihood approach and least squares; see lecture

September 11: presentation of possible projects

September 16: basic formulas of probabilistic uncertainty; see paper 17-88

September 18: review for Test 1

September 23: Test 1

September 25: Monte-Carlo algorithm for propagating probabilistic uncertainty, see montecarlo slides

September 30: review of the results of Test 1

October 2: linearization and interval case; slides 1-11 of slides atlanta18, and Section 6 of paper 07-53b; see also linearization slides

October 7: work on your projects day

October 9: work on your projects day

October 14: how to program interval propagation algorithms; see lecture on bisection and lecture on programming numerical differentiation

October 16: uncertainty propagation: fuzzy case; slides 56-57 of slides atlanta18

October 21: uncertainty propagation via AI algorithms: paper 24-31 and Section 2 of paper23-66a

October 23: reasoning under uncertainty; slides slides kazan19 and slides 1-60 from fuzzySlides

October 28: rehearsal of project presentations at UTEP/NMSU workshop

October 30: decision making under interval and fuzzy uncertainty; see lecture on decision making under interval uncertainty, Section 4 of paper 12-33a, and Slides 6-15, 44-45, 23-25, and 87-92 from slides ssci18 and slides 81-90 from fuzzySlides

November 4: symmetry approach to decision making; see paper 25-21 and related slides

November 6: symmetry approach to decision making (cont-d); see paper 19-49

November 11: symmetry approach to decision making (cont-d); see paper 19-49, paper tr19-105b, and slides aici20.pdf

November 13: uncertainty visualization; see Section 8 of the paper 25-07

November 18: review for Test 2

November 20: Test 2

November 25: project presentations

December 2: overview of the results of Test 2

December 4: review for the final exam